2024-08-16T02:25:21.122 INFO:root:teuthology version: 1.2.0 2024-08-16T02:25:21.138 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:25:21.217 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419 branch: wip-yingxin-testing-2024-08-15-1551 description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7857419' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-testing-2024-08-15-1551 ceph: conf: client: debug ms: 1 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 workunit: branch: wip-yingxin-testing-2024-08-15-1551 env: CRIMSON_COMPAT: '1' sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-testing-2024-08-15-1551 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgJy+r4YXXHKqjf+iPZ773bSWUgJiAtywUiTZ3yPFMtIyOAaT9jQ0/D5LRRWKk6rYOqLh+6v2TyadFXMDVIN+M= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJN0eLkAaQKXqliGcXiWf+GWaN8aDOqj1a33CqLWfDhU0S0tMZG/K2as32jjpaHd/R4tCg76H4QPgNsz9vK9jY= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rados/test.sh --crimson - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-16_01:59:29 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-16T02:25:21.217 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa; will attempt to use it 2024-08-16T02:25:21.218 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks 2024-08-16T02:25:21.218 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-16T02:25:21.219 INFO:teuthology.task.internal:Checking packages... 2024-08-16T02:25:21.240 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '6f386a3f0b1fe6a4d7219b54bf963d790bc84004' 2024-08-16T02:25:21.240 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-16T02:25:21.240 INFO:teuthology.packaging:ref: None 2024-08-16T02:25:21.240 INFO:teuthology.packaging:tag: None 2024-08-16T02:25:21.240 INFO:teuthology.packaging:branch: wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:25:21.240 INFO:teuthology.packaging:sha1: 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:25:21.241 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:25:21.376 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5812.g6f386a3f 2024-08-16T02:25:21.378 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-16T02:25:21.386 INFO:teuthology.task.internal:no buildpackages task found 2024-08-16T02:25:21.386 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-16T02:25:21.425 INFO:teuthology.task.internal:Saving configuration 2024-08-16T02:25:21.438 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-16T02:25:21.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-16T02:25:21.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:20:08.505299', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgJy+r4YXXHKqjf+iPZ773bSWUgJiAtywUiTZ3yPFMtIyOAaT9jQ0/D5LRRWKk6rYOqLh+6v2TyadFXMDVIN+M='} 2024-08-16T02:25:21.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-16 02:20:08.506502', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJN0eLkAaQKXqliGcXiWf+GWaN8aDOqj1a33CqLWfDhU0S0tMZG/K2as32jjpaHd/R4tCg76H4QPgNsz9vK9jY='} 2024-08-16T02:25:21.489 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-16T02:25:21.496 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-08-16T02:25:21.496 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-08-16T02:25:21.496 INFO:teuthology.run_tasks:Running task console_log... 2024-08-16T02:25:21.579 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fef586b3ac0>, signals=[15]) 2024-08-16T02:25:21.579 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-16T02:25:21.596 INFO:teuthology.task.internal:Opening connections... 2024-08-16T02:25:21.596 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:25:21.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:25:21.679 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:25:21.680 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:25:21.758 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-16T02:25:21.763 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-08-16T02:25:21.782 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-08-16T02:25:21.782 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-08-16T02:25:21.839 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="CentOS Stream" 2024-08-16T02:25:21.839 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="9" 2024-08-16T02:25:21.839 INFO:teuthology.orchestra.run.smithi116.stdout:ID="centos" 2024-08-16T02:25:21.839 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:25:21.839 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="9" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:ANSI_COLOR="0;31" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:25:21.840 INFO:teuthology.orchestra.run.smithi116.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:25:21.841 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-08-16T02:25:21.870 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-08-16T02:25:21.888 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-08-16T02:25:21.889 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-08-16T02:25:21.946 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-08-16T02:25:21.946 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-08-16T02:25:21.946 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-08-16T02:25:21.946 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-08-16T02:25:21.946 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-16T02:25:21.947 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-16T02:25:21.948 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-08-16T02:25:21.978 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-16T02:25:21.985 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-16T02:25:22.008 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-16T02:25:22.009 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:25:22.011 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-08-16T02:25:22.028 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-16T02:25:22.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-16T02:25:22.033 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:25:22.068 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-08-16T02:25:22.167 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-16T02:25:22.211 INFO:teuthology.run_tasks:Running task kernel... 2024-08-16T02:25:22.229 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-08-16T02:25:22.229 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-08-16T02:25:22.230 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:25:22.230 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-16T02:25:22.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:25:22.247 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-08-16T02:25:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:25:22.248 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-08-16T02:25:22.304 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:25:22.304 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.14.0-437.el9.x86_64 2024-08-16T02:25:22.305 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum install -y kernel 2024-08-16T02:25:22.306 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-437.el9.x86_64 2024-08-16T02:25:22.306 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-437.el9.x86_64 2024-08-16T02:25:22.306 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-08-16T02:25:24.487 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-08-16T02:25:24.747 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-08-16T02:25:26.598 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-08-16T02:25:29.406 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 20 MB 00:03 2024-08-16T02:25:30.964 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.6 MB 00:00 2024-08-16T02:25:32.623 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 18 kB 00:00 2024-08-16T02:25:33.760 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-16T02:25:33.931 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.6 MB 00:00 2024-08-16T02:25:35.824 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 18 kB 00:00 2024-08-16T02:25:36.686 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-08-16T02:25:39.133 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-16T02:25:41.272 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:25:41.273 INFO:teuthology.orchestra.run.smithi116.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:25:41.373 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:25:41.383 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:Install 4 Packages 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 84 M 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 126 M 2024-08-16T02:25:41.384 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:25:42.025 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-08-16T02:25:42.534 INFO:teuthology.orchestra.run.smithi116.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.5 MB/s | 1.5 MB 00:01 2024-08-16T02:25:42.976 INFO:teuthology.orchestra.run.smithi116.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2024-08-16T02:25:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-08-16T02:25:43.990 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-16T02:25:43.990 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-16T02:25:44.083 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-16T02:25:44.088 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:Install 4 Packages 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 84 M 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 126 M 2024-08-16T02:25:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:25:44.627 INFO:teuthology.orchestra.run.smithi152.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 3.0 MB/s | 1.5 MB 00:00 2024-08-16T02:25:45.434 INFO:teuthology.orchestra.run.smithi116.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 10 MB/s | 30 MB 00:02 2024-08-16T02:25:45.434 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:25:45.435 INFO:teuthology.orchestra.run.smithi116.stdout:Total 21 MB/s | 84 MB 00:04 2024-08-16T02:25:46.168 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:25:46.509 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:25:46.509 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:25:47.018 INFO:teuthology.orchestra.run.smithi152.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 5.9 MB/s | 17 MB 00:02 2024-08-16T02:25:47.506 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:25:47.507 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:25:48.585 INFO:teuthology.orchestra.run.smithi152.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 8.1 MB/s | 36 MB 00:04 2024-08-16T02:25:49.227 INFO:teuthology.orchestra.run.smithi152.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 6.5 MB/s | 30 MB 00:04 2024-08-16T02:25:49.227 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:25:49.227 INFO:teuthology.orchestra.run.smithi152.stdout:Total 16 MB/s | 84 MB 00:05 2024-08-16T02:25:49.961 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:25:49.972 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:25:50.301 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:25:50.302 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:25:50.726 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:25:50.834 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:25:51.308 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:25:51.308 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:25:53.508 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:25:53.644 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:25:53.761 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:25:54.413 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:25:54.513 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:25:57.176 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:25:57.324 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:25:58.500 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:25:58.561 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:02.213 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:02.342 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:03.492 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:07.226 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:33.312 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:33.352 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:33.727 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:33.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:33.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:33.727 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:26:34.503 INFO:teuthology.orchestra.run.smithi116.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:34.504 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:26:34.504 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:26:34.901 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:34.985 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:34.986 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:34.986 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:26:34.986 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:26:34.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:26:34.987 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-16T02:26:34.987 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:26:34.987 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:35.044 INFO:teuthology.orchestra.run.smithi116.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:35.045 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:35.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo rpm -qi grub2-tools 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Name : grub2-tools 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Epoch : 1 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Version : 2.06 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Release : 68.el9 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: x86_64 2024-08-16T02:26:35.077 INFO:teuthology.orchestra.run.smithi116.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Group : Unspecified 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Size : 8264254 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:License : GPLv3+ 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Packager : builder@centos.org 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Vendor : CentOS 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Summary : Support tools for GRUB. 2024-08-16T02:26:35.078 INFO:teuthology.orchestra.run.smithi116.stdout:Description : 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout:hardware devices. 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:26:35.079 INFO:teuthology.orchestra.run.smithi116.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:26:35.080 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:26:35.080 INFO:teuthology.task.kernel:Updating grub on smithi116 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:26:35.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:26:35.653 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-08-16T02:26:37.061 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:37.064 INFO:teuthology.orchestra.run.smithi116.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:26:37.097 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-08-16T02:26:37.101 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:37.102 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-08-16T02:26:37.117 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.gdRDJaWQiK 2024-08-16T02:26:37.118 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gdRDJaWQiK 2024-08-16T02:26:37.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.gdRDJaWQiK 2024-08-16T02:26:37.336 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.gdRDJaWQiK is 6KB 2024-08-16T02:26:37.357 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.gdRDJaWQiK 2024-08-16T02:26:37.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:26:37.432 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:26:37.432 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:26:37.432 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:26:37.432 INFO:teuthology.orchestra.run.smithi116.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:26:37.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:26:37.484 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:37.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-16T02:26:37.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-16T02:26:37.484 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-16T02:26:37.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-08-16T02:26:38.325 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-16T02:26:38.325 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:26:38.325 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:26:38.326 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:26:38.710 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:38.799 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.799 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:38.799 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-16T02:26:38.799 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-16T02:26:38.800 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-16T02:26:38.800 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-08-16T02:26:38.800 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2024-08-16T02:26:38.800 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-08-16T02:26:38.857 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-16T02:26:38.857 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-16T02:26:38.857 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2024-08-16T02:26:38.890 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2024-08-16T02:26:38.890 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2024-08-16T02:26:38.890 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.06 2024-08-16T02:26:38.890 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 68.el9 2024-08-16T02:26:38.890 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Group : Unspecified 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 8264254 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : builder@centos.org 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2024-08-16T02:26:38.891 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:26:38.892 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2024-08-16T02:26:38.893 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-16T02:26:38.893 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 5.14.0-480.el9.x86_64 2024-08-16T02:26:38.893 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-16T02:26:39.502 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2024-08-16T02:26:40.916 INFO:teuthology.orchestra.run.smithi152.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-16T02:26:40.948 INFO:teuthology.orchestra.run.smithi152.stderr:done 2024-08-16T02:26:40.950 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-08-16T02:26:40.965 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.9osv3QbYpo 2024-08-16T02:26:40.965 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9osv3QbYpo 2024-08-16T02:26:41.031 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.9osv3QbYpo 2024-08-16T02:26:41.167 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.9osv3QbYpo is 6KB 2024-08-16T02:26:41.216 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.9osv3QbYpo 2024-08-16T02:26:41.229 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2024-08-16T02:26:41.291 INFO:teuthology.orchestra.run.smithi152.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-16T02:26:41.291 INFO:teuthology.orchestra.run.smithi152.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-16T02:26:41.291 INFO:teuthology.orchestra.run.smithi152.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-16T02:26:41.291 INFO:teuthology.orchestra.run.smithi152.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-16T02:26:41.293 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-16T02:26:41.434 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2024-08-16T02:27:07.610 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-16T02:27:07.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:27:07.612 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:11.440 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-08-16T02:27:11.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:27:11.441 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:26.047 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-08-16T02:27:29.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-08-16T02:27:35.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:27:35.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:38.111 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-08-16T02:27:38.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:27:38.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:41.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2024-08-16T02:27:50.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:27:50.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:27:53.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:27:53.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:28:22.494 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-08-16T02:28:22.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:28:22.968 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:28:22.968 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-08-16T02:28:23.007 INFO:teuthology.orchestra.run.smithi116.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:28:23.007 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:28:23.007 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:28:23.008 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:28:24.008 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-16T02:28:24.009 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:28:24.068 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-08-16T02:28:24.106 DEBUG:teuthology.parallel:result is None 2024-08-16T02:28:53.962 DEBUG:teuthology.orchestra.remote:timed out 2024-08-16T02:29:08.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:29:08.966 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:29:09.369 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-08-16T02:29:09.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:29:09.777 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-480.el9.x86_64"... 2024-08-16T02:29:09.778 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-08-16T02:29:09.796 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-480.el9.x86_64 2024-08-16T02:29:09.796 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-16T02:29:09.796 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-16T02:29:09.796 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-16T02:29:10.797 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-08-16T02:29:10.798 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-16T02:29:10.864 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-08-16T02:29:10.898 DEBUG:teuthology.parallel:result is None 2024-08-16T02:29:10.898 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-16T02:29:10.905 INFO:teuthology.task.internal:Creating test directory... 2024-08-16T02:29:10.905 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:29:10.908 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-16T02:29:10.927 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-16T02:29:10.953 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-16T02:29:10.986 INFO:teuthology.task.internal:Creating archive directory... 2024-08-16T02:29:10.986 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:29:10.989 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-16T02:29:11.101 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-16T02:29:11.110 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-16T02:29:11.110 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:29:11.113 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-16T02:29:11.143 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:29:11.145 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:29:11.154 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:29:11.157 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-16T02:29:11.159 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-16T02:29:11.165 INFO:teuthology.task.internal:Configuring sudo... 2024-08-16T02:29:11.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:29:11.199 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-16T02:29:11.227 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-16T02:29:11.237 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-16T02:29:11.237 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:29:11.267 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:29:11.284 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:29:11.367 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:29:11.421 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:29:11.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:29:11.545 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:29:11.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:29:11.610 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:29:11.659 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-16T02:29:11.704 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:29:11.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-16T02:29:11.832 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:29:11.832 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-16T02:29:11.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-08-16T02:29:11.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-08-16T02:29:11.946 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:29:11.982 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:29:12.421 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-16T02:29:12.429 INFO:teuthology.task.internal:Starting timer... 2024-08-16T02:29:12.429 INFO:teuthology.run_tasks:Running task pcp... 2024-08-16T02:29:12.456 INFO:teuthology.run_tasks:Running task selinux... 2024-08-16T02:29:12.489 DEBUG:teuthology.orchestra.run.smithi116:> sudo service auditd rotate 2024-08-16T02:29:12.533 INFO:teuthology.orchestra.run.smithi116.stdout:Rotating logs: 2024-08-16T02:29:12.535 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-08-16T02:29:12.586 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-08-16T02:29:12.588 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-16T02:29:12.588 DEBUG:teuthology.orchestra.run.smithi116:> /usr/sbin/getenforce 2024-08-16T02:29:12.670 INFO:teuthology.orchestra.run.smithi116.stdout:Permissive 2024-08-16T02:29:12.670 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-08-16T02:29:12.698 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-08-16T02:29:12.698 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi116.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-08-16T02:29:12.698 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:29:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:29:12.728 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:29:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:29:12.766 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-16T02:29:12.767 DEBUG:teuthology.orchestra.run.smithi116:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:29:12.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-08-16T02:29:12.850 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-16T02:29:12.860 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-16T02:29:12.871 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-16T02:29:12.872 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-16T02:33:45.340 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-08-16T02:33:45.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:33:45.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:33:45.420 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-08-16T02:33:45.502 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-08-16T02:33:45.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:33:45.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-16T02:33:45.579 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-08-16T02:33:45.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-08-16T02:33:45.663 INFO:teuthology.run_tasks:Running task clock... 2024-08-16T02:33:45.674 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-16T02:33:45.674 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:33:45.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:33:45.677 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-16T02:33:45.677 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:33:45.703 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:33:45.717 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:33:45.735 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-16T02:33:45.741 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: ntpd: command not found 2024-08-16T02:33:45.750 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-16T02:33:45.751 INFO:teuthology.orchestra.run.smithi116.stdout:506 Cannot talk to daemon 2024-08-16T02:33:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:33:45.776 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-08-16T02:33:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:33:45.787 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-08-16T02:33:45.802 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-16T02:33:45.817 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-16T02:33:45.832 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:33:45.835 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:33:45.835 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-08-16T02:33:45.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.835 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.836 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.872 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.875 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:33:45.876 INFO:teuthology.run_tasks:Running task install... 2024-08-16T02:33:45.884 DEBUG:teuthology.task.install:project ceph 2024-08-16T02:33:45.885 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}} 2024-08-16T02:33:45.885 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'} 2024-08-16T02:33:45.885 INFO:teuthology.task.install:Using flavor: crimson 2024-08-16T02:33:45.891 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-16T02:33:45.891 INFO:teuthology.task.install:extra packages: [] 2024-08-16T02:33:45.891 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:33:45.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:33:45.894 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'tag': None, 'wait_for_package': False} 2024-08-16T02:33:45.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:33:46.038 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:33:46.038 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:33:46.055 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/ 2024-08-16T02:33:46.055 INFO:teuthology.task.install.rpm:Package version is 19.0.0-5812.g6f386a3f 2024-08-16T02:33:46.147 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:33:46.148 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:33:46.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:33:46.163 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yingxin-testing-2024-08-15-1551/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-16T02:33:46.164 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:33:46.164 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-16T02:33:46.173 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:33:46.173 DEBUG:teuthology.orchestra.run.smithi116:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:33:46.189 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-16T02:33:46.189 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6f386a3f0b1fe6a4d7219b54bf963d790bc84004/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-16T02:33:46.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:33:46.255 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-16T02:33:46.314 DEBUG:teuthology.orchestra.run.smithi116:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:33:46.330 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-16T02:33:46.378 INFO:teuthology.orchestra.run.smithi116.stdout:check_obsoletes = 1 2024-08-16T02:33:46.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum clean all 2024-08-16T02:33:46.394 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-08-16T02:33:46.396 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-08-16T02:33:46.605 INFO:teuthology.orchestra.run.smithi116.stdout:45 files removed 2024-08-16T02:33:46.629 INFO:teuthology.orchestra.run.smithi152.stdout:45 files removed 2024-08-16T02:33:46.632 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-radosgw 2024-08-16T02:33:46.656 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-08-16T02:33:47.231 INFO:teuthology.orchestra.run.smithi116.stdout:ceph packages for x86_64 272 kB/s | 88 kB 00:00 2024-08-16T02:33:47.243 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 284 kB/s | 88 kB 00:00 2024-08-16T02:33:47.582 INFO:teuthology.orchestra.run.smithi116.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-08-16T02:33:47.594 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 80 kB/s | 24 kB 00:00 2024-08-16T02:33:47.903 INFO:teuthology.orchestra.run.smithi116.stdout:ceph source packages 7.6 kB/s | 2.1 kB 00:00 2024-08-16T02:33:47.911 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 7.6 kB/s | 2.1 kB 00:00 2024-08-16T02:33:48.445 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.2 MB 00:00 2024-08-16T02:33:49.874 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-08-16T02:33:52.750 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-08-16T02:33:54.113 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-08-16T02:33:57.051 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-16T02:33:59.343 INFO:teuthology.orchestra.run.smithi116.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-16T02:33:59.713 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.6 MB 00:01 2024-08-16T02:34:00.481 INFO:teuthology.orchestra.run.smithi116.stdout:Extra Packages for Enterprise Linux 22 MB/s | 22 MB 00:01 2024-08-16T02:34:01.539 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-08-16T02:34:02.494 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-16T02:34:05.886 INFO:teuthology.orchestra.run.smithi116.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-08-16T02:34:07.825 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-16T02:34:08.220 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrading: 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:34:08.224 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:34:08.225 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:34:08.226 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout:Installing weak dependencies: 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout:Install 30 Packages 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout:Upgrade 2 Packages 2024-08-16T02:34:08.227 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:08.228 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 60 M 2024-08-16T02:34:08.228 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:34:10.088 INFO:teuthology.orchestra.run.smithi116.stdout:(1/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 4.3 MB/s | 6.1 MB 00:01 2024-08-16T02:34:10.129 INFO:teuthology.orchestra.run.smithi116.stdout:(2/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 604 kB/s | 25 kB 00:00 2024-08-16T02:34:10.142 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:34:10.156 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 10 M 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.8 M 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.1 M 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 20 M 2024-08-16T02:34:10.157 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 25 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 807 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 558 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.4 M 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-16T02:34:10.158 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 45 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 138 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 185 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 364 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 357 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 117 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-16T02:34:10.159 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout:Install 30 Packages 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-08-16T02:34:10.160 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:10.161 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 60 M 2024-08-16T02:34:10.161 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:34:10.230 INFO:teuthology.orchestra.run.smithi116.stdout:(3/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 7.9 MB/s | 807 kB 00:00 2024-08-16T02:34:10.288 INFO:teuthology.orchestra.run.smithi116.stdout:(4/32): libradosstriper1-19.0.0-5812.g6f386a3f. 9.4 MB/s | 558 kB 00:00 2024-08-16T02:34:10.630 INFO:teuthology.orchestra.run.smithi116.stdout:(5/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_6 16 MB/s | 5.4 MB 00:00 2024-08-16T02:34:10.689 INFO:teuthology.orchestra.run.smithi116.stdout:(6/32): python3-ceph-argparse-19.0.0-5812.g6f38 775 kB/s | 45 kB 00:00 2024-08-16T02:34:10.898 INFO:teuthology.orchestra.run.smithi116.stdout:(7/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 8.9 MB/s | 20 MB 00:02 2024-08-16T02:34:10.922 INFO:teuthology.orchestra.run.smithi116.stdout:(8/32): python3-ceph-common-19.0.0-5812.g6f386a 593 kB/s | 138 kB 00:00 2024-08-16T02:34:10.940 INFO:teuthology.orchestra.run.smithi116.stdout:(9/32): python3-cephfs-19.0.0-5812.g6f386a3f.el 4.3 MB/s | 185 kB 00:00 2024-08-16T02:34:11.057 INFO:teuthology.orchestra.run.smithi116.stdout:(10/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9 4.4 MB/s | 10 MB 00:02 2024-08-16T02:34:11.090 INFO:teuthology.orchestra.run.smithi116.stdout:(11/32): python3-rados-19.0.0-5812.g6f386a3f.el 2.1 MB/s | 364 kB 00:00 2024-08-16T02:34:11.115 INFO:teuthology.orchestra.run.smithi116.stdout:(12/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 2.0 MB/s | 357 kB 00:00 2024-08-16T02:34:11.132 INFO:teuthology.orchestra.run.smithi116.stdout:(13/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 1.5 MB/s | 117 kB 00:00 2024-08-16T02:34:11.258 INFO:teuthology.orchestra.run.smithi116.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2024-08-16T02:34:11.274 INFO:teuthology.orchestra.run.smithi116.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 581 kB/s | 107 kB 00:00 2024-08-16T02:34:11.291 INFO:teuthology.orchestra.run.smithi116.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 911 kB/s | 160 kB 00:00 2024-08-16T02:34:11.342 INFO:teuthology.orchestra.run.smithi116.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-08-16T02:34:11.367 INFO:teuthology.orchestra.run.smithi116.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-08-16T02:34:11.392 INFO:teuthology.orchestra.run.smithi116.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-08-16T02:34:11.418 INFO:teuthology.orchestra.run.smithi116.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 712 kB/s | 36 kB 00:00 2024-08-16T02:34:11.443 INFO:teuthology.orchestra.run.smithi116.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 936 kB/s | 47 kB 00:00 2024-08-16T02:34:11.468 INFO:teuthology.orchestra.run.smithi116.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-08-16T02:34:11.577 INFO:teuthology.orchestra.run.smithi116.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-16T02:34:11.602 INFO:teuthology.orchestra.run.smithi116.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-16T02:34:11.628 INFO:teuthology.orchestra.run.smithi116.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 244 kB/s | 26 kB 00:00 2024-08-16T02:34:11.653 INFO:teuthology.orchestra.run.smithi116.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-16T02:34:11.678 INFO:teuthology.orchestra.run.smithi116.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-08-16T02:34:11.720 INFO:teuthology.orchestra.run.smithi116.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-08-16T02:34:11.770 INFO:teuthology.orchestra.run.smithi116.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-08-16T02:34:11.863 INFO:teuthology.orchestra.run.smithi116.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-16T02:34:11.867 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-common-19.0.0-5812.g6f386a3f.el9.x 16 MB/s | 20 MB 00:01 2024-08-16T02:34:11.951 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-base-19.0.0-5812.g6f386a3f.el9.x86 4.6 MB/s | 6.1 MB 00:01 2024-08-16T02:34:11.976 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): ceph-selinux-19.0.0-5812.g6f386a3f.el9. 231 kB/s | 25 kB 00:00 2024-08-16T02:34:11.979 INFO:teuthology.orchestra.run.smithi116.stdout:(31/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 13 MB/s | 3.5 MB 00:00 2024-08-16T02:34:12.054 INFO:teuthology.orchestra.run.smithi116.stdout:(32/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 13 MB/s | 3.8 MB 00:00 2024-08-16T02:34:12.059 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:12.059 INFO:teuthology.orchestra.run.smithi116.stdout:Total 16 MB/s | 60 MB 00:03 2024-08-16T02:34:12.085 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): ceph-radosgw-19.0.0-5812.g6f386a3f.el9. 7.2 MB/s | 10 MB 00:01 2024-08-16T02:34:12.119 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): libcephfs2-19.0.0-5812.g6f386a3f.el9.x8 4.7 MB/s | 807 kB 00:00 2024-08-16T02:34:12.152 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): libradosstriper1-19.0.0-5812.g6f386a3f. 3.1 MB/s | 558 kB 00:00 2024-08-16T02:34:12.154 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:34:12.178 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): python3-ceph-argparse-19.0.0-5812.g6f38 767 kB/s | 45 kB 00:00 2024-08-16T02:34:12.190 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:34:12.190 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:34:12.203 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-ceph-common-19.0.0-5812.g6f386a 2.7 MB/s | 138 kB 00:00 2024-08-16T02:34:12.237 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-cephfs-19.0.0-5812.g6f386a3f.el 3.1 MB/s | 185 kB 00:00 2024-08-16T02:34:12.262 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): python3-rados-19.0.0-5812.g6f386a3f.el 6.1 MB/s | 364 kB 00:00 2024-08-16T02:34:12.296 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-rbd-19.0.0-5812.g6f386a3f.el9. 6.0 MB/s | 357 kB 00:00 2024-08-16T02:34:12.313 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): python3-rgw-19.0.0-5812.g6f386a3f.el9. 2.3 MB/s | 117 kB 00:00 2024-08-16T02:34:12.397 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): librgw2-19.0.0-5812.g6f386a3f.el9.x86_ 17 MB/s | 5.4 MB 00:00 2024-08-16T02:34:12.505 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 419 kB/s | 45 kB 00:00 2024-08-16T02:34:12.530 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 457 kB/s | 107 kB 00:00 2024-08-16T02:34:12.556 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 160 kB 00:00 2024-08-16T02:34:12.631 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 563 kB/s | 42 kB 00:00 2024-08-16T02:34:12.682 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-08-16T02:34:12.723 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-08-16T02:34:12.748 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 538 kB/s | 36 kB 00:00 2024-08-16T02:34:12.771 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:34:12.771 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:34:12.808 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-08-16T02:34:12.833 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 434 kB/s | 47 kB 00:00 2024-08-16T02:34:12.916 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-08-16T02:34:12.941 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-16T02:34:12.967 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-08-16T02:34:12.992 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-16T02:34:13.017 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.5 MB/s | 192 kB 00:00 2024-08-16T02:34:13.076 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-16T02:34:13.118 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-16T02:34:13.202 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-16T02:34:13.344 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): librbd1-19.0.0-5812.g6f386a3f.el9.x86_ 17 MB/s | 3.8 MB 00:00 2024-08-16T02:34:13.452 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): librados2-19.0.0-5812.g6f386a3f.el9.x8 9.2 MB/s | 3.5 MB 00:00 2024-08-16T02:34:13.457 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:13.457 INFO:teuthology.orchestra.run.smithi152.stdout:Total 18 MB/s | 60 MB 00:03 2024-08-16T02:34:13.555 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:34:13.591 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:34:13.591 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:34:14.170 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:34:14.170 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:34:14.177 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:34:14.698 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:34:14.748 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:14.868 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:14.955 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:14.983 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:15.046 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:15.099 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:34:15.129 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:34:15.163 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:34:15.186 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:34:15.216 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:34:15.363 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:34:15.403 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:34:15.601 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:34:15.765 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:34:15.823 INFO:teuthology.orchestra.run.smithi116.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:15.881 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:15.926 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:34:15.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:16.012 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:34:16.091 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:34:16.109 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:34:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-16T02:34:16.199 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:16.205 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:34:16.249 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:34:16.287 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:16.357 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:34:16.381 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:16.392 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:34:16.417 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:16.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:34:16.463 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:16.473 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:34:16.502 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:34:16.506 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-08-16T02:34:16.538 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-08-16T02:34:16.541 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:34:16.564 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-08-16T02:34:16.578 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:34:16.586 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:34:16.611 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-08-16T02:34:16.806 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:34:16.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-08-16T02:34:17.162 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:34:17.190 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:34:17.199 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-08-16T02:34:17.249 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:17.254 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:34:17.299 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:17.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:34:17.343 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 12/34 2024-08-16T02:34:17.350 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:34:17.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:17.438 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 14/34 2024-08-16T02:34:17.526 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-08-16T02:34:17.552 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:34:17.633 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 16/34 2024-08-16T02:34:17.667 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-08-16T02:34:17.775 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-08-16T02:34:17.827 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 19/34 2024-08-16T02:34:17.875 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-08-16T02:34:17.925 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-08-16T02:34:17.955 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 22/34 2024-08-16T02:34:17.986 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-08-16T02:34:18.022 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-16T02:34:18.640 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-16T02:34:18.667 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:34:18.741 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 26/34 2024-08-16T02:34:18.819 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 27/34 2024-08-16T02:34:18.862 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 28/34 2024-08-16T02:34:19.805 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:19.848 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:20.547 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:20.585 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:34:20.666 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:34:20.666 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:34:20.667 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:34:20.667 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:20.694 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:21.342 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:21.376 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:22.080 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 29/34 2024-08-16T02:34:22.122 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:34:22.222 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 30/34 2024-08-16T02:34:22.222 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-16T02:34:22.222 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-16T02:34:22.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:22.248 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /sys 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /proc 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /mnt 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /var/tmp 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /home 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /root 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout:skipping the directory /tmp 2024-08-16T02:34:30.735 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:31.849 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:31.874 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:31.876 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:31.959 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:31.960 INFO:teuthology.orchestra.run.smithi116.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:32.554 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:32.554 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-08-16T02:34:32.555 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:34:33.207 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:34:33.208 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:34:33.209 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:34:33.210 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:34:33.643 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:33.669 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 32/34 2024-08-16T02:34:33.669 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:34:33.670 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-16T02:34:33.670 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:33.670 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-16T02:34:33.670 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:33.672 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:33.737 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-16T02:34:33.738 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:33.958 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:33.958 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout:Upgraded: 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:34:33.959 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.960 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:33.961 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:34:34.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-test 2024-08-16T02:34:34.869 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:29 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:34:34.995 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:34:34.996 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout:Install 9 Packages 2024-08-16T02:34:34.997 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:34.998 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 88 M 2024-08-16T02:34:34.998 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 834 M 2024-08-16T02:34:34.998 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_6 6/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 7/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9. 8/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x8 9/34 2024-08-16T02:34:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 10/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 11/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 12/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 13/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-16T02:34:35.064 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-16T02:34:35.065 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-16T02:34:35.066 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-16T02:34:35.066 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 31/34 2024-08-16T02:34:35.066 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-16T02:34:35.066 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/34 2024-08-16T02:34:35.588 INFO:teuthology.orchestra.run.smithi116.stdout:(1/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 593 kB/s | 175 kB 00:00 2024-08-16T02:34:35.813 INFO:teuthology.orchestra.run.smithi116.stdout:(2/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 2.0 MB/s | 469 kB 00:00 2024-08-16T02:34:35.903 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.904 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-16T02:34:35.905 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-16T02:34:35.906 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:35.907 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:34:35.989 INFO:teuthology.orchestra.run.smithi116.stdout:(3/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 135 kB/s | 94 kB 00:00 2024-08-16T02:34:36.014 INFO:teuthology.orchestra.run.smithi116.stdout:(4/9): libubsan-11.4.1-3.el9.x86_64.rpm 933 kB/s | 187 kB 00:00 2024-08-16T02:34:36.222 INFO:teuthology.orchestra.run.smithi116.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-08-16T02:34:36.274 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-08-16T02:34:36.281 INFO:teuthology.orchestra.run.smithi116.stdout:(6/9): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-08-16T02:34:36.340 INFO:teuthology.orchestra.run.smithi116.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.0 MB/s | 122 kB 00:00 2024-08-16T02:34:36.382 INFO:teuthology.orchestra.run.smithi116.stdout:(8/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 401 kB/s | 64 kB 00:00 2024-08-16T02:34:36.829 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:29 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:34:36.955 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 86 M 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: libasan8 x86_64 13.3.1-2.1.el9 appstream 469 k 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 175 k 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-08-16T02:34:36.956 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout:Install 9 Packages 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:34:36.957 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 88 M 2024-08-16T02:34:36.958 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 834 M 2024-08-16T02:34:36.958 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:34:37.796 INFO:teuthology.orchestra.run.smithi152.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 745 kB/s | 94 kB 00:00 2024-08-16T02:34:37.840 INFO:teuthology.orchestra.run.smithi116.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 34 MB/s | 86 MB 00:02 2024-08-16T02:34:37.846 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:37.846 INFO:teuthology.orchestra.run.smithi116.stdout:Total 31 MB/s | 88 MB 00:02 2024-08-16T02:34:37.926 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:34:37.946 INFO:teuthology.orchestra.run.smithi152.stdout:(2/9): libcephsqlite-19.0.0-5812.g6f386a3f.el9. 632 kB/s | 175 kB 00:00 2024-08-16T02:34:37.948 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:34:37.948 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:34:37.996 INFO:teuthology.orchestra.run.smithi152.stdout:(3/9): libasan8-13.3.1-2.1.el9.x86_64.rpm 2.3 MB/s | 469 kB 00:00 2024-08-16T02:34:38.097 INFO:teuthology.orchestra.run.smithi152.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-08-16T02:34:38.122 INFO:teuthology.orchestra.run.smithi152.stdout:(5/9): libubsan-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-08-16T02:34:38.147 INFO:teuthology.orchestra.run.smithi152.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 6.0 MB/s | 304 kB 00:00 2024-08-16T02:34:38.197 INFO:teuthology.orchestra.run.smithi152.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 846 kB/s | 64 kB 00:00 2024-08-16T02:34:38.222 INFO:teuthology.orchestra.run.smithi152.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.6 MB/s | 122 kB 00:00 2024-08-16T02:34:38.552 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:34:38.553 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:34:39.453 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:34:39.522 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:34:39.574 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:34:39.627 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:34:39.654 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:34:39.686 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:39.710 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:39.788 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:39.836 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:34:39.862 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:40.323 INFO:teuthology.orchestra.run.smithi152.stdout:(9/9): ceph-test-19.0.0-5812.g6f386a3f.el9.x86_ 32 MB/s | 86 MB 00:02 2024-08-16T02:34:40.326 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:34:40.326 INFO:teuthology.orchestra.run.smithi152.stdout:Total 26 MB/s | 88 MB 00:03 2024-08-16T02:34:40.410 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:34:40.432 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:34:40.432 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:34:41.040 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:34:41.041 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:34:41.926 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:34:42.022 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-08-16T02:34:42.066 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-08-16T02:34:42.111 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-08-16T02:34:42.137 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-08-16T02:34:42.176 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:42.201 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:42.278 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libasan8-13.3.1-2.1.el9.x86_64 6/9 2024-08-16T02:34:42.319 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-08-16T02:34:42.345 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:51.431 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:51.519 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:53.927 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 8/9 2024-08-16T02:34:54.012 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:34:57.846 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:34:58.174 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:34:58.175 INFO:teuthology.orchestra.run.smithi116.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:34:58.176 INFO:teuthology.orchestra.run.smithi116.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:34:58.176 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:58.176 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:34:58.488 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph 2024-08-16T02:34:59.044 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:34:59.173 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:34:59.176 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:34:59.177 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:34:59.178 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:34:59.179 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:34:59.180 INFO:teuthology.orchestra.run.smithi116.stdout:Install 37 Packages 2024-08-16T02:34:59.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:34:59.180 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 86 M 2024-08-16T02:34:59.180 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 943 M 2024-08-16T02:34:59.180 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:34:59.787 INFO:teuthology.orchestra.run.smithi116.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-08-16T02:35:00.004 INFO:teuthology.orchestra.run.smithi116.stdout:(2/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 5.7 MB/s | 2.2 MB 00:00 2024-08-16T02:35:00.046 INFO:teuthology.orchestra.run.smithi116.stdout:(3/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 3.7 MB/s | 1.6 MB 00:00 2024-08-16T02:35:00.171 INFO:teuthology.orchestra.run.smithi116.stdout:(4/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86_ 13 MB/s | 5.1 MB 00:00 2024-08-16T02:35:00.196 INFO:teuthology.orchestra.run.smithi116.stdout:(5/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 1.8 MB/s | 284 kB 00:00 2024-08-16T02:35:00.354 INFO:teuthology.orchestra.run.smithi116.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-08-16T02:35:00.472 INFO:teuthology.orchestra.run.smithi116.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 4.5 MB/s | 1.2 MB 00:00 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 9/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libasan8-13.3.1-2.1.el9.x86_64 4/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-08-16T02:35:00.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-08-16T02:35:00.530 INFO:teuthology.orchestra.run.smithi116.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 608 kB/s | 106 kB 00:00 2024-08-16T02:35:00.614 INFO:teuthology.orchestra.run.smithi116.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 953 kB/s | 135 kB 00:00 2024-08-16T02:35:00.714 INFO:teuthology.orchestra.run.smithi116.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 680 kB/s | 125 kB 00:00 2024-08-16T02:35:00.789 INFO:teuthology.orchestra.run.smithi116.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.2 MB/s | 215 kB 00:00 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: libasan8-13.3.1-2.1.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: libubsan-11.4.1-3.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-16T02:35:00.897 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-16T02:35:00.898 INFO:teuthology.orchestra.run.smithi152.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-08-16T02:35:00.898 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:00.898 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:01.180 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-08-16T02:35:01.223 INFO:teuthology.orchestra.run.smithi116.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 370 kB/s | 188 kB 00:00 2024-08-16T02:35:01.290 INFO:teuthology.orchestra.run.smithi116.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 344 kB/s | 172 kB 00:00 2024-08-16T02:35:01.323 INFO:teuthology.orchestra.run.smithi116.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-08-16T02:35:01.734 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:54 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:01.864 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:01.866 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:01.866 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:35:01.866 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:01.866 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:35:01.866 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 6.5 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 2.2 M 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 1.6 M 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 284 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 5.1 M 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 72 M 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-16T02:35:01.867 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-16T02:35:01.868 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-16T02:35:01.869 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout:Install 37 Packages 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 86 M 2024-08-16T02:35:01.870 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 943 M 2024-08-16T02:35:01.871 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:35:01.957 INFO:teuthology.orchestra.run.smithi116.stdout:(15/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86 37 MB/s | 72 MB 00:01 2024-08-16T02:35:01.974 INFO:teuthology.orchestra.run.smithi116.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 78 kB/s | 51 kB 00:00 2024-08-16T02:35:01.999 INFO:teuthology.orchestra.run.smithi116.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 109 kB/s | 77 kB 00:00 2024-08-16T02:35:02.066 INFO:teuthology.orchestra.run.smithi116.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-08-16T02:35:02.091 INFO:teuthology.orchestra.run.smithi116.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-08-16T02:35:02.117 INFO:teuthology.orchestra.run.smithi116.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-08-16T02:35:02.167 INFO:teuthology.orchestra.run.smithi116.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-08-16T02:35:02.193 INFO:teuthology.orchestra.run.smithi116.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 473 kB/s | 11 kB 00:00 2024-08-16T02:35:02.218 INFO:teuthology.orchestra.run.smithi116.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-08-16T02:35:02.243 INFO:teuthology.orchestra.run.smithi116.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 924 kB/s | 23 kB 00:00 2024-08-16T02:35:02.268 INFO:teuthology.orchestra.run.smithi116.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 781 kB/s | 19 kB 00:00 2024-08-16T02:35:02.294 INFO:teuthology.orchestra.run.smithi116.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-08-16T02:35:02.319 INFO:teuthology.orchestra.run.smithi116.stdout:(27/37): python3-toml-0.10.2-6.el9.noarch.rpm 115 kB/s | 42 kB 00:00 2024-08-16T02:35:02.339 INFO:teuthology.orchestra.run.smithi152.stdout:(1/37): ceph-19.0.0-5812.g6f386a3f.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2024-08-16T02:35:02.344 INFO:teuthology.orchestra.run.smithi116.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-08-16T02:35:02.394 INFO:teuthology.orchestra.run.smithi116.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-08-16T02:35:02.420 INFO:teuthology.orchestra.run.smithi116.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 724 kB/s | 16 kB 00:00 2024-08-16T02:35:02.445 INFO:teuthology.orchestra.run.smithi116.stdout:(31/37): python3-more-itertools-8.12.0-2.el9.no 627 kB/s | 79 kB 00:00 2024-08-16T02:35:02.470 INFO:teuthology.orchestra.run.smithi116.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-16T02:35:02.495 INFO:teuthology.orchestra.run.smithi116.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-08-16T02:35:02.512 INFO:teuthology.orchestra.run.smithi116.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-08-16T02:35:02.529 INFO:teuthology.orchestra.run.smithi116.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-16T02:35:02.555 INFO:teuthology.orchestra.run.smithi116.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-08-16T02:35:02.579 INFO:teuthology.orchestra.run.smithi116.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-08-16T02:35:02.584 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:02.584 INFO:teuthology.orchestra.run.smithi116.stdout:Total 25 MB/s | 86 MB 00:03 2024-08-16T02:35:02.737 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:02.772 INFO:teuthology.orchestra.run.smithi152.stdout:(2/37): ceph-mgr-19.0.0-5812.g6f386a3f.el9.x86_ 2.6 MB/s | 1.6 MB 00:00 2024-08-16T02:35:02.779 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:02.779 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:02.856 INFO:teuthology.orchestra.run.smithi152.stdout:(3/37): ceph-mds-19.0.0-5812.g6f386a3f.el9.x86_ 3.2 MB/s | 2.2 MB 00:00 2024-08-16T02:35:02.923 INFO:teuthology.orchestra.run.smithi152.stdout:(4/37): ceph-mgr-modules-core-19.0.0-5812.g6f38 4.2 MB/s | 284 kB 00:00 2024-08-16T02:35:03.065 INFO:teuthology.orchestra.run.smithi152.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-08-16T02:35:03.240 INFO:teuthology.orchestra.run.smithi152.stdout:(6/37): ceph-mon-19.0.0-5812.g6f386a3f.el9.x86_ 5.6 MB/s | 5.1 MB 00:00 2024-08-16T02:35:03.315 INFO:teuthology.orchestra.run.smithi152.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 4.9 MB/s | 1.2 MB 00:00 2024-08-16T02:35:03.349 INFO:teuthology.orchestra.run.smithi152.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 983 kB/s | 106 kB 00:00 2024-08-16T02:35:03.399 INFO:teuthology.orchestra.run.smithi152.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.6 MB/s | 135 kB 00:00 2024-08-16T02:35:03.424 INFO:teuthology.orchestra.run.smithi152.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-08-16T02:35:03.483 INFO:teuthology.orchestra.run.smithi152.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-08-16T02:35:03.566 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:03.566 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:03.925 INFO:teuthology.orchestra.run.smithi152.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 376 kB/s | 188 kB 00:00 2024-08-16T02:35:03.984 INFO:teuthology.orchestra.run.smithi152.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 344 kB/s | 172 kB 00:00 2024-08-16T02:35:04.017 INFO:teuthology.orchestra.run.smithi152.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 379 kB/s | 35 kB 00:00 2024-08-16T02:35:04.084 INFO:teuthology.orchestra.run.smithi152.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 770 kB/s | 77 kB 00:00 2024-08-16T02:35:04.110 INFO:teuthology.orchestra.run.smithi152.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-08-16T02:35:04.185 INFO:teuthology.orchestra.run.smithi152.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 414 kB/s | 42 kB 00:00 2024-08-16T02:35:04.202 INFO:teuthology.orchestra.run.smithi152.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 242 kB/s | 22 kB 00:00 2024-08-16T02:35:04.277 INFO:teuthology.orchestra.run.smithi152.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 581 kB/s | 43 kB 00:00 2024-08-16T02:35:04.311 INFO:teuthology.orchestra.run.smithi152.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-08-16T02:35:04.337 INFO:teuthology.orchestra.run.smithi152.stdout:(21/37): luarocks-3.9.2-5.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-08-16T02:35:04.353 INFO:teuthology.orchestra.run.smithi152.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 641 kB/s | 11 kB 00:00 2024-08-16T02:35:04.379 INFO:teuthology.orchestra.run.smithi152.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 707 kB/s | 18 kB 00:00 2024-08-16T02:35:04.404 INFO:teuthology.orchestra.run.smithi152.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 935 kB/s | 23 kB 00:00 2024-08-16T02:35:04.438 INFO:teuthology.orchestra.run.smithi152.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-08-16T02:35:04.463 INFO:teuthology.orchestra.run.smithi152.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-08-16T02:35:04.488 INFO:teuthology.orchestra.run.smithi152.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 785 kB/s | 20 kB 00:00 2024-08-16T02:35:04.513 INFO:teuthology.orchestra.run.smithi152.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-08-16T02:35:04.539 INFO:teuthology.orchestra.run.smithi152.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-16T02:35:04.564 INFO:teuthology.orchestra.run.smithi152.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-08-16T02:35:04.589 INFO:teuthology.orchestra.run.smithi152.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-08-16T02:35:04.615 INFO:teuthology.orchestra.run.smithi152.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-08-16T02:35:04.640 INFO:teuthology.orchestra.run.smithi152.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 707 kB/s | 36 kB 00:00 2024-08-16T02:35:04.665 INFO:teuthology.orchestra.run.smithi152.stdout:(34/37): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-16T02:35:04.699 INFO:teuthology.orchestra.run.smithi152.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 597 kB/s | 20 kB 00:00 2024-08-16T02:35:04.732 INFO:teuthology.orchestra.run.smithi152.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-08-16T02:35:05.299 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:05.350 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:35:05.382 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:35:05.410 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:35:05.482 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:35:05.526 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:35:05.600 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:35:05.694 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:35:05.771 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:35:05.811 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:35:05.846 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:35:05.876 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:35:05.910 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:35:05.944 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:35:05.984 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:35:06.027 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:35:06.033 INFO:teuthology.orchestra.run.smithi152.stdout:(37/37): ceph-osd-19.0.0-5812.g6f386a3f.el9.x86 22 MB/s | 72 MB 00:03 2024-08-16T02:35:06.037 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:06.037 INFO:teuthology.orchestra.run.smithi152.stdout:Total 21 MB/s | 86 MB 00:04 2024-08-16T02:35:06.079 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:35:06.127 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:35:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:35:06.188 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:35:06.231 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:35:06.231 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:35:06.233 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:35:06.371 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:35:06.451 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:35:06.512 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:35:06.550 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:35:06.608 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:35:06.663 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:35:06.850 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:35:06.905 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:35:06.983 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:35:07.006 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:35:07.007 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:35:07.071 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:35:07.112 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:35:07.248 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:35:07.440 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:35:07.464 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:35:07.490 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:35:07.490 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:07.490 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:35:07.490 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:35:07.491 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:35:07.491 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:08.702 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:35:08.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-16T02:35:08.804 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-16T02:35:08.838 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-16T02:35:08.909 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-16T02:35:08.954 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-16T02:35:08.987 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-16T02:35:09.079 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-16T02:35:09.157 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-16T02:35:09.206 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-16T02:35:09.241 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-16T02:35:09.271 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-16T02:35:09.306 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-16T02:35:09.339 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-16T02:35:09.380 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-16T02:35:09.440 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-16T02:35:09.474 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-16T02:35:09.522 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-16T02:35:09.563 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-16T02:35:09.636 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-16T02:35:09.785 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-16T02:35:09.871 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-16T02:35:09.924 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-16T02:35:09.971 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-16T02:35:10.021 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-16T02:35:10.076 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-16T02:35:10.263 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-16T02:35:10.326 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-16T02:35:10.403 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-16T02:35:10.492 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-16T02:35:10.533 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-16T02:35:10.660 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-16T02:35:10.856 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 32/37 2024-08-16T02:35:10.885 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:35:10.911 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 33/37 2024-08-16T02:35:10.912 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:10.912 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-16T02:35:10.912 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:35:10.912 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-16T02:35:10.912 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:18.480 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:35:18.507 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:19.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:35:19.063 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:35:19.063 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:19.063 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:35:19.064 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:35:19.064 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:35:19.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:19.290 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:35:19.313 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:35:19.313 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:19.314 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:35:19.314 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:35:19.314 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:35:19.314 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:19.387 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:35:21.994 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 34/37 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-16T02:35:22.021 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:22.561 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 35/37 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-16T02:35:22.585 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:22.812 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:35:22.835 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 36/37 2024-08-16T02:35:22.835 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:22.835 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-16T02:35:22.836 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:35:22.836 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-16T02:35:22.836 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:22.900 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:35:26.184 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:35:26.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:35:26.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:35:26.184 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:35:26.185 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:35:26.186 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:35:26.187 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:35:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:35:27.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:27.170 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:35:27.171 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:35:27.172 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:35:27.173 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:27.174 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:27.432 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-base 2024-08-16T02:35:28.017 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:23 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:28.106 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:28.150 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:28.151 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:35:28.151 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:28.209 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install cephadm 2024-08-16T02:35:28.764 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:23 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:28.891 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:28.892 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:28.892 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:35:28.892 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:28.892 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 773 k 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 779 k 2024-08-16T02:35:28.893 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:35:29.173 INFO:teuthology.orchestra.run.smithi116.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.7 MB/s | 773 kB 00:00 2024-08-16T02:35:29.173 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:29.173 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.7 MB/s | 773 kB 00:00 2024-08-16T02:35:29.173 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:29.179 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:29.179 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:29.184 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:29.185 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:29.257 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:29.725 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:29.800 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 37/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 3/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 4/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 5/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9. 6/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-16T02:35:30.158 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-16T02:35:30.159 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-16T02:35:30.160 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-16T02:35:30.160 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-16T02:35:30.160 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-16T02:35:30.160 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-16T02:35:30.161 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-16T02:35:30.162 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-16T02:35:30.162 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-16T02:35:30.162 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-16T02:35:30.162 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-16T02:35:30.162 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-16T02:35:30.173 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:30.423 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:30.424 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:30.424 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:30.424 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:30.634 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:35:31.196 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:26 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:31.268 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-16T02:35:31.269 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-16T02:35:31.270 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:31.271 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:31.327 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:31.328 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 158 k 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 644 k 2024-08-16T02:35:31.329 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:35:31.544 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 734 kB/s | 158 kB 00:00 2024-08-16T02:35:31.545 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:31.545 INFO:teuthology.orchestra.run.smithi116.stdout:Total 730 kB/s | 158 kB 00:00 2024-08-16T02:35:31.545 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:31.555 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:31.555 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:31.566 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-08-16T02:35:31.611 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:31.612 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:31.761 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:31.828 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:31.853 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:35:31.853 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:32.118 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:25 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:32.205 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:32.246 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:32.247 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:35:32.247 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:32.305 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-08-16T02:35:32.712 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:32.713 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:32.713 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:32.713 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:32.713 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:32.713 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:32.853 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:25 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:32.956 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr 2024-08-16T02:35:32.981 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:32.981 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 773 k 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 773 k 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 779 k 2024-08-16T02:35:32.982 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:35:33.287 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.0.0-5812.g6f386a3f.el9.noarch.rpm 2.5 MB/s | 773 kB 00:00 2024-08-16T02:35:33.288 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:33.288 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.5 MB/s | 773 kB 00:00 2024-08-16T02:35:33.288 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:35:33.293 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:35:33.294 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:35:33.299 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:35:33.300 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:35:33.370 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:35:33.503 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:28 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:33.594 INFO:teuthology.orchestra.run.smithi116.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:33.635 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:33.637 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:35:33.637 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:33.699 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:35:33.820 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:33.910 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:34.243 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:29 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:34.316 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:34.369 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:35:34.370 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout:Install 11 Packages 2024-08-16T02:35:34.371 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:34.372 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 18 M 2024-08-16T02:35:34.372 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 154 M 2024-08-16T02:35:34.372 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:35:34.582 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:35:34.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:34.583 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:35:34.583 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:34.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:34.583 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:34.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-08-16T02:35:35.273 INFO:teuthology.orchestra.run.smithi116.stdout:(1/11): ceph-grafana-dashboards-19.0.0-5812.g6f 175 kB/s | 34 kB 00:00 2024-08-16T02:35:35.298 INFO:teuthology.orchestra.run.smithi116.stdout:(2/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 74 kB/s | 16 kB 00:00 2024-08-16T02:35:35.363 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:28 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:35.456 INFO:teuthology.orchestra.run.smithi116.stdout:(3/11): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-08-16T02:35:35.492 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:35.493 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:35.493 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-08-16T02:35:35.493 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:35.493 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 158 k 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 158 k 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 644 k 2024-08-16T02:35:35.494 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:35:35.532 INFO:teuthology.orchestra.run.smithi116.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.2 MB/s | 548 kB 00:00 2024-08-16T02:35:35.582 INFO:teuthology.orchestra.run.smithi116.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 751 kB/s | 19 kB 00:00 2024-08-16T02:35:35.649 INFO:teuthology.orchestra.run.smithi116.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-08-16T02:35:35.674 INFO:teuthology.orchestra.run.smithi116.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.9 MB/s | 144 kB 00:00 2024-08-16T02:35:35.699 INFO:teuthology.orchestra.run.smithi116.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-16T02:35:35.713 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.0.0-5812.g6f386a 722 kB/s | 158 kB 00:00 2024-08-16T02:35:35.713 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:35.714 INFO:teuthology.orchestra.run.smithi152.stdout:Total 717 kB/s | 158 kB 00:00 2024-08-16T02:35:35.714 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:35:35.723 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:35:35.723 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:35:35.725 INFO:teuthology.orchestra.run.smithi116.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-08-16T02:35:35.780 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:35:35.780 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:35:35.866 INFO:teuthology.orchestra.run.smithi116.stdout:(10/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a 18 MB/s | 14 MB 00:00 2024-08-16T02:35:35.935 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:35:35.997 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:36.021 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:36.022 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:35:36.022 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-16T02:35:36.022 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:36.175 INFO:teuthology.orchestra.run.smithi116.stdout:(11/11): python3-protobuf-3.14.0-13.el9.noarch. 298 kB/s | 269 kB 00:00 2024-08-16T02:35:36.179 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:36.179 INFO:teuthology.orchestra.run.smithi116.stdout:Total 10 MB/s | 18 MB 00:01 2024-08-16T02:35:36.209 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:36.228 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:36.228 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:36.459 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:36.459 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:36.932 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f. 1/1 2024-08-16T02:35:36.933 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:36.933 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:35:36.933 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:35:36.933 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:36.933 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:36.995 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:37.073 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:35:37.103 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:35:37.161 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-08-16T02:35:37.203 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:35:37.337 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:35:37.398 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:35:37.482 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:35:37.526 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:35:37.554 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:37.586 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:35:37.721 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:30 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:37.810 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:35:37.853 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:37.854 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:35:37.854 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:37.914 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-08-16T02:35:38.465 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:31 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:38.591 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 14 M 2024-08-16T02:35:38.592 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 34 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 16 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:38.593 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:35:38.594 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:35:38.594 INFO:teuthology.orchestra.run.smithi152.stdout:Install 11 Packages 2024-08-16T02:35:38.594 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:38.594 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 18 M 2024-08-16T02:35:38.594 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 154 M 2024-08-16T02:35:38.595 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:35:38.958 INFO:teuthology.orchestra.run.smithi152.stdout:(1/11): ceph-prometheus-alerts-19.0.0-5812.g6f3 88 kB/s | 16 kB 00:00 2024-08-16T02:35:39.133 INFO:teuthology.orchestra.run.smithi152.stdout:(2/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-08-16T02:35:39.400 INFO:teuthology.orchestra.run.smithi152.stdout:(3/11): ceph-mgr-dashboard-19.0.0-5812.g6f386a3 23 MB/s | 14 MB 00:00 2024-08-16T02:35:39.492 INFO:teuthology.orchestra.run.smithi152.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.9 MB/s | 548 kB 00:00 2024-08-16T02:35:39.509 INFO:teuthology.orchestra.run.smithi152.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-08-16T02:35:39.576 INFO:teuthology.orchestra.run.smithi152.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-08-16T02:35:39.601 INFO:teuthology.orchestra.run.smithi152.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-08-16T02:35:39.626 INFO:teuthology.orchestra.run.smithi152.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-08-16T02:35:39.652 INFO:teuthology.orchestra.run.smithi152.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-08-16T02:35:39.860 INFO:teuthology.orchestra.run.smithi152.stdout:(10/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.2 MB/s | 863 kB 00:00 2024-08-16T02:35:40.747 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:35:40.808 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:40.960 INFO:teuthology.orchestra.run.smithi152.stdout:(11/11): ceph-grafana-dashboards-19.0.0-5812.g6 16 kB/s | 34 kB 00:02 2024-08-16T02:35:40.963 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:40.964 INFO:teuthology.orchestra.run.smithi152.stdout:Total 7.7 MB/s | 18 MB 00:02 2024-08-16T02:35:40.997 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:35:41.019 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:35:41.019 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:35:41.257 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:35:41.258 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:35:41.802 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:35:41.901 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-08-16T02:35:41.930 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-08-16T02:35:42.026 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-08-16T02:35:42.174 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-08-16T02:35:42.225 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-08-16T02:35:42.309 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-08-16T02:35:42.362 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-08-16T02:35:42.389 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:42.421 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 9/11 2024-08-16T02:35:42.577 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:42.577 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:35:42.577 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:35:42.577 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:35:42.577 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:35:42.578 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:35:42.879 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:35:42.879 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:35:42.880 INFO:teuthology.orchestra.run.smithi116.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:35:42.881 INFO:teuthology.orchestra.run.smithi116.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:35:42.881 INFO:teuthology.orchestra.run.smithi116.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:35:42.881 INFO:teuthology.orchestra.run.smithi116.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:35:42.881 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:42.881 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:43.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:35:43.705 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:38 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:43.833 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:43.834 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:35:43.835 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout:======================================================================================= 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout:Install 12 Packages 2024-08-16T02:35:43.836 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:43.837 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 38 M 2024-08-16T02:35:43.837 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 209 M 2024-08-16T02:35:43.837 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:35:44.365 INFO:teuthology.orchestra.run.smithi116.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-08-16T02:35:44.448 INFO:teuthology.orchestra.run.smithi116.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-08-16T02:35:44.473 INFO:teuthology.orchestra.run.smithi116.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 301 kB/s | 33 kB 00:00 2024-08-16T02:35:44.524 INFO:teuthology.orchestra.run.smithi116.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-08-16T02:35:44.582 INFO:teuthology.orchestra.run.smithi116.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 604 kB/s | 35 kB 00:00 2024-08-16T02:35:44.683 INFO:teuthology.orchestra.run.smithi116.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-58 15 MB/s | 7.4 MB 00:00 2024-08-16T02:35:44.833 INFO:teuthology.orchestra.run.smithi116.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-08-16T02:35:44.908 INFO:teuthology.orchestra.run.smithi116.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2024-08-16T02:35:44.933 INFO:teuthology.orchestra.run.smithi116.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 976 kB/s | 244 kB 00:00 2024-08-16T02:35:44.958 INFO:teuthology.orchestra.run.smithi116.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 4.9 MB/s | 250 kB 00:00 2024-08-16T02:35:45.059 INFO:teuthology.orchestra.run.smithi116.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-08-16T02:35:45.359 INFO:teuthology.orchestra.run.smithi116.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-08-16T02:35:45.360 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:45.360 INFO:teuthology.orchestra.run.smithi116.stdout:Total 25 MB/s | 38 MB 00:01 2024-08-16T02:35:45.606 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 10/11 2024-08-16T02:35:45.683 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:45.711 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:45.737 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:45.738 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:46.164 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:46.165 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:46.960 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:47.039 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:35:47.082 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:47.180 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:47.216 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:35:47.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:35:47.433 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:35:47.509 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:35:47.630 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 11/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el 1/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noa 2/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9 3/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-08-16T02:35:47.631 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-08-16T02:35:47.948 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-16T02:35:47.949 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-16T02:35:47.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-16T02:35:47.950 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:47.950 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:35:48.203 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:35:48.213 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-16T02:35:48.321 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:35:48.759 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:41 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:35:48.888 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 7.4 M 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:35:48.889 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:48.890 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 38 M 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-08-16T02:35:48.891 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:35:49.448 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 887 kB/s | 195 kB 00:00 2024-08-16T02:35:49.544 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:49.573 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-08-16T02:35:49.690 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): ceph-mgr-diskprediction-local-19.0.0-58 16 MB/s | 7.4 MB 00:00 2024-08-16T02:35:50.074 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 52 kB/s | 33 kB 00:00 2024-08-16T02:35:50.215 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 36 kB/s | 18 kB 00:00 2024-08-16T02:35:50.580 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:35:50.615 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:50.665 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 60 kB/s | 35 kB 00:00 2024-08-16T02:35:50.732 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-08-16T02:35:51.374 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.1 MB/s | 4.7 MB 00:01 2024-08-16T02:35:51.466 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 305 kB/s | 244 kB 00:00 2024-08-16T02:35:51.891 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.4 MB/s | 5.1 MB 00:01 2024-08-16T02:35:52.191 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 307 kB/s | 250 kB 00:00 2024-08-16T02:35:52.967 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-08-16T02:35:52.968 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:52.969 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.3 MB/s | 38 MB 00:04 2024-08-16T02:35:53.322 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:35:53.350 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:35:53.350 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:35:53.789 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:35:53.789 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:35:53.826 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:35:53.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:35:53.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:35:53.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:35:53.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:53.827 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:35:54.196 INFO:teuthology.orchestra.run.smithi116.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:54.197 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:54.541 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:35:54.609 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-16T02:35:54.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:35:54.648 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:35:54.748 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:35:54.782 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-16T02:35:54.961 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-16T02:35:54.989 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-16T02:35:55.066 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-16T02:35:55.193 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:50 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:35:55.320 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:35:55.322 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout:Install 13 Packages 2024-08-16T02:35:55.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:55.324 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 2.2 M 2024-08-16T02:35:55.324 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 26 M 2024-08-16T02:35:55.324 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:35:55.599 INFO:teuthology.orchestra.run.smithi116.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 229 kB/s | 19 kB 00:00 2024-08-16T02:35:55.632 INFO:teuthology.orchestra.run.smithi116.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 26 kB 00:00 2024-08-16T02:35:55.716 INFO:teuthology.orchestra.run.smithi116.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-08-16T02:35:55.723 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-16T02:35:55.741 INFO:teuthology.orchestra.run.smithi116.stdout:(4/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 218 kB/s | 49 kB 00:00 2024-08-16T02:35:55.766 INFO:teuthology.orchestra.run.smithi116.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-08-16T02:35:55.800 INFO:teuthology.orchestra.run.smithi116.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.2 MB/s | 279 kB 00:00 2024-08-16T02:35:55.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-16T02:35:55.850 INFO:teuthology.orchestra.run.smithi116.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 494 kB/s | 54 kB 00:00 2024-08-16T02:35:55.900 INFO:teuthology.orchestra.run.smithi116.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-08-16T02:35:55.951 INFO:teuthology.orchestra.run.smithi116.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 112 kB/s | 14 kB 00:00 2024-08-16T02:35:55.984 INFO:teuthology.orchestra.run.smithi116.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.9 MB/s | 59 kB 00:00 2024-08-16T02:35:56.018 INFO:teuthology.orchestra.run.smithi116.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-08-16T02:35:56.043 INFO:teuthology.orchestra.run.smithi116.stdout:(12/13): python3-google-auth-2.33.0-1.el9.noarc 1.3 MB/s | 220 kB 00:00 2024-08-16T02:35:56.077 INFO:teuthology.orchestra.run.smithi116.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-08-16T02:35:56.079 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:35:56.079 INFO:teuthology.orchestra.run.smithi116.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-08-16T02:35:56.141 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:35:56.156 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:35:56.156 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:35:56.242 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:35:56.242 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:35:56.749 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:35:56.929 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:35:56.983 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:35:57.038 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:35:57.084 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:35:57.104 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:35:57.124 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:35:57.227 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:35:57.337 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:35:57.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:35:58.111 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:35:58.155 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-16T02:35:58.158 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:35:58.184 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:35:58.187 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:35:58.232 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:35:58.254 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:35:59.123 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:35:59.124 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:35:59.124 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:35:59.445 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:35:59.446 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:35:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:35:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:35:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:35:59.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:35:59.447 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:35:59.717 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:36:00.274 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:55 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:00.403 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:36:00.404 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout:Install 6 Packages 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:00.405 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 7.0 M 2024-08-16T02:36:00.406 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 33 M 2024-08-16T02:36:00.406 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:00.879 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 12/12 2024-08-16T02:36:00.879 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386 1/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-16T02:36:00.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-16T02:36:01.038 INFO:teuthology.orchestra.run.smithi116.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-08-16T02:36:01.088 INFO:teuthology.orchestra.run.smithi116.stdout:(2/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 674 kB/s | 150 kB 00:00 2024-08-16T02:36:01.147 INFO:teuthology.orchestra.run.smithi116.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-08-16T02:36:01.180 INFO:teuthology.orchestra.run.smithi116.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.3 MB/s | 74 kB 00:00 2024-08-16T02:36:01.206 INFO:teuthology.orchestra.run.smithi116.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 500 kB/s | 58 kB 00:00 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-16T02:36:01.277 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:01.278 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:01.289 INFO:teuthology.orchestra.run.smithi116.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-08-16T02:36:01.292 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:01.292 INFO:teuthology.orchestra.run.smithi116.stdout:Total 7.9 MB/s | 7.0 MB 00:00 2024-08-16T02:36:01.362 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:01.373 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:01.373 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:01.471 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:01.472 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:01.764 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-08-16T02:36:01.779 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:01.952 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:36:02.008 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:36:02.321 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:55 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:02.450 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 49 k 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-16T02:36:02.451 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.33.0-1.el9 epel 220 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:02.452 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.2 M 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-08-16T02:36:02.453 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:02.520 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:36:02.603 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:36:02.669 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:36:02.696 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:36:02.790 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 26 kB 00:00 2024-08-16T02:36:02.815 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 19 kB 00:00 2024-08-16T02:36:02.840 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): ceph-mgr-rook-19.0.0-5812.g6f386a3f.el9 256 kB/s | 49 kB 00:00 2024-08-16T02:36:02.958 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-08-16T02:36:02.983 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 951 kB/s | 159 kB 00:00 2024-08-16T02:36:03.033 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 712 kB/s | 54 kB 00:00 2024-08-16T02:36:03.083 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2024-08-16T02:36:03.117 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-08-16T02:36:03.142 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 132 kB/s | 14 kB 00:00 2024-08-16T02:36:03.193 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-08-16T02:36:03.218 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-google-auth-2.33.0-1.el9.noarc 1.6 MB/s | 220 kB 00:00 2024-08-16T02:36:03.243 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-08-16T02:36:03.285 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-08-16T02:36:03.288 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:03.288 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-08-16T02:36:03.351 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:03.366 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:03.366 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:03.458 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:03.458 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:36:03.606 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:36:03.877 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:36:03.877 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:03.877 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:03.877 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:03.877 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:36:03.878 INFO:teuthology.orchestra.run.smithi116.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:36:03.878 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:36:03.878 INFO:teuthology.orchestra.run.smithi116.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:36:03.878 INFO:teuthology.orchestra.run.smithi116.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:36:03.878 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:03.879 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:03.976 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:04.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-fuse 2024-08-16T02:36:04.141 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-16T02:36:04.196 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-16T02:36:04.250 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-16T02:36:04.288 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-16T02:36:04.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-16T02:36:04.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-16T02:36:04.550 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.33.0-1.el9.noarch 7/13 2024-08-16T02:36:04.606 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-16T02:36:04.678 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:01:59 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:04.810 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Installing dependencies: 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:36:04.811 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:Install 2 Packages 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 963 k 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 4.1 M 2024-08-16T02:36:04.812 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:05.228 INFO:teuthology.orchestra.run.smithi116.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 562 kB/s | 80 kB 00:00 2024-08-16T02:36:05.330 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-16T02:36:05.380 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-16T02:36:05.395 INFO:teuthology.orchestra.run.smithi116.stdout:(2/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 2.8 MB/s | 883 kB 00:00 2024-08-16T02:36:05.395 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:05.395 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.6 MB/s | 963 kB 00:00 2024-08-16T02:36:05.407 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:05.413 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-16T02:36:05.430 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:05.430 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:05.461 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-16T02:36:05.483 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:36:05.508 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:05.509 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:05.697 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:05.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:36:05.912 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:36:06.270 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 13/13 2024-08-16T02:36:06.270 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 1/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.33.0-1.el9.noarch 10/13 2024-08-16T02:36:06.271 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-16T02:36:06.272 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-16T02:36:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:36:06.560 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-16T02:36:06.628 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.33.0-1.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:06.629 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:06.801 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:06.897 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-08-16T02:36:07.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install ceph-volume 2024-08-16T02:36:07.448 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:00 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:07.577 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 150 k 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-16T02:36:07.578 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-08-16T02:36:07.579 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:07.580 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.0 M 2024-08-16T02:36:07.580 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-08-16T02:36:07.580 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:07.639 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:02 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:07.773 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:07.773 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repository Size 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:07.774 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 285 k 2024-08-16T02:36:07.775 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 1.3 M 2024-08-16T02:36:07.775 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:07.881 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-08-16T02:36:07.948 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-08-16T02:36:07.973 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): ceph-mgr-cephadm-19.0.0-5812.g6f386a3f.e 646 kB/s | 150 kB 00:00 2024-08-16T02:36:07.998 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-16T02:36:08.015 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-08-16T02:36:08.015 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:08.015 INFO:teuthology.orchestra.run.smithi116.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-08-16T02:36:08.016 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:08.024 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:08.024 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:08.052 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:08.052 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:08.065 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 811 kB/s | 74 kB 00:00 2024-08-16T02:36:08.271 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:08.317 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:08.332 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:08.332 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:36:08.332 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:36:08.333 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:09.224 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-08-16T02:36:09.227 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:09.227 INFO:teuthology.orchestra.run.smithi152.stdout:Total 4.3 MB/s | 7.0 MB 00:01 2024-08-16T02:36:09.302 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:09.314 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:09.314 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:09.325 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:09.414 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:09.414 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:09.627 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados-devel 2024-08-16T02:36:09.715 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:09.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-16T02:36:09.942 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-16T02:36:10.202 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:05 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:10.331 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:10.331 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:10.331 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-16T02:36:10.331 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 125 k 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 445 k 2024-08-16T02:36:10.332 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:10.460 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-16T02:36:10.544 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-16T02:36:10.580 INFO:teuthology.orchestra.run.smithi116.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 505 kB/s | 125 kB 00:00 2024-08-16T02:36:10.580 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:10.581 INFO:teuthology.orchestra.run.smithi116.stdout:Total 502 kB/s | 125 kB 00:00 2024-08-16T02:36:10.581 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:10.586 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:10.586 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:10.610 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-16T02:36:10.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:36:10.651 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:10.651 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:10.814 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:10.918 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:11.402 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:11.504 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 6/6 2024-08-16T02:36:11.504 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 1/6 2024-08-16T02:36:11.505 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-16T02:36:11.505 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-16T02:36:11.505 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-16T02:36:11.505 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-16T02:36:11.682 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:11.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:11.683 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:11.683 INFO:teuthology.orchestra.run.smithi116.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:11.683 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:11.683 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:11.768 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-16T02:36:11.768 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:11.769 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:11.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs2 2024-08-16T02:36:11.997 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-08-16T02:36:12.536 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:07 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:12.544 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:05 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:12.624 INFO:teuthology.orchestra.run.smithi116.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:12.669 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:12.756 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 883 k 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-08-16T02:36:12.757 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:12.758 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 963 k 2024-08-16T02:36:12.758 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 4.1 M 2024-08-16T02:36:12.758 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:12.758 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:12.758 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:12.758 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install libcephfs-devel 2024-08-16T02:36:13.001 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 301 kB/s | 80 kB 00:00 2024-08-16T02:36:13.051 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-fuse-19.0.0-5812.g6f386a3f.el9.x86_ 2.7 MB/s | 883 kB 00:00 2024-08-16T02:36:13.052 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:13.052 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.5 MB/s | 963 kB 00:00 2024-08-16T02:36:13.063 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:13.086 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:13.086 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:13.165 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:13.165 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:13.308 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:08 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:13.351 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:13.437 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:13.438 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:13.439 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 32 k 2024-08-16T02:36:13.439 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 142 k 2024-08-16T02:36:13.439 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:13.489 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-16T02:36:13.553 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:36:13.635 INFO:teuthology.orchestra.run.smithi116.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 162 kB/s | 32 kB 00:00 2024-08-16T02:36:13.635 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:13.635 INFO:teuthology.orchestra.run.smithi116.stdout:Total 161 kB/s | 32 kB 00:00 2024-08-16T02:36:13.635 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:13.639 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:13.639 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:13.653 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:13.653 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:13.735 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:13.840 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:14.217 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2/2 2024-08-16T02:36:14.217 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/2 2024-08-16T02:36:14.247 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:14.508 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:14.511 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:14.685 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librados2 2024-08-16T02:36:14.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-08-16T02:36:15.229 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:10 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:15.314 INFO:teuthology.orchestra.run.smithi116.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:15.357 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:15.359 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:15.359 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:15.364 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:08 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:15.423 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install librbd1 2024-08-16T02:36:15.497 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-5812.g6f386a3f.el9 ceph-noarch 285 k 2024-08-16T02:36:15.498 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 285 k 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.3 M 2024-08-16T02:36:15.499 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:15.744 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-volume-19.0.0-5812.g6f386a3f.el9.noarch.rp 1.1 MB/s | 285 kB 00:00 2024-08-16T02:36:15.745 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:15.745 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-08-16T02:36:15.745 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:15.753 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:15.753 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:15.781 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:15.781 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:15.978 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:10 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:15.999 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:16.066 INFO:teuthology.orchestra.run.smithi116.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:16.071 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:16.086 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:16.087 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:36:16.087 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-16T02:36:16.087 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:16.109 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:16.110 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:16.111 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:16.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rados 2024-08-16T02:36:16.730 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:11 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:16.815 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:16.859 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:16.860 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:16.860 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:16.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rgw 2024-08-16T02:36:17.110 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 1/1 2024-08-16T02:36:17.110 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:17.111 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:17.111 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-5812.g6f386a3f.el9.noarch 2024-08-16T02:36:17.111 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:17.111 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:17.397 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-08-16T02:36:17.481 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:12 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:17.566 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:17.609 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:17.610 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:17.610 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:17.677 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-cephfs 2024-08-16T02:36:17.974 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:10 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:18.103 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:18.103 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 125 k 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:18.104 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:18.105 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 125 k 2024-08-16T02:36:18.105 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 445 k 2024-08-16T02:36:18.105 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:18.226 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:13 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:18.311 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:18.353 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:18.354 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:18.354 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:18.417 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install python3-rbd 2024-08-16T02:36:18.978 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:13 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:19.064 INFO:teuthology.orchestra.run.smithi116.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:19.106 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:19.107 INFO:teuthology.orchestra.run.smithi116.stdout:Nothing to do. 2024-08-16T02:36:19.107 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:19.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-fuse 2024-08-16T02:36:19.733 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:14 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:19.865 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-16T02:36:19.866 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:19.867 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:19.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:19.868 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 91 k 2024-08-16T02:36:19.868 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 321 k 2024-08-16T02:36:19.868 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:20.071 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 446 kB/s | 91 kB 00:00 2024-08-16T02:36:20.071 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:20.072 INFO:teuthology.orchestra.run.smithi116.stdout:Total 442 kB/s | 91 kB 00:00 2024-08-16T02:36:20.072 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:20.081 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:20.081 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:20.145 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:20.146 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:20.289 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:20.394 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:20.515 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.0.0-5812.g6f386a3f.el9.x86_64 52 kB/s | 125 kB 00:02 2024-08-16T02:36:20.516 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:20.516 INFO:teuthology.orchestra.run.smithi152.stdout:Total 52 kB/s | 125 kB 00:02 2024-08-16T02:36:20.516 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:20.523 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:20.523 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:20.590 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:20.591 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:20.757 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:20.760 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:20.854 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:20.999 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:21.000 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:21.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:21.000 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:21.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-mirror 2024-08-16T02:36:21.294 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:21.575 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:21.811 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:16 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:21.851 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-08-16T02:36:21.945 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout: Package Arch Version Repo Size 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:21.946 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 3.5 M 2024-08-16T02:36:21.947 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 30 M 2024-08-16T02:36:21.947 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 9.8 MB/s | 3.5 MB 00:00 2024-08-16T02:36:22.303 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stdout:Total 9.7 MB/s | 3.5 MB 00:00 2024-08-16T02:36:22.304 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:22.315 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:22.315 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:22.400 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:22.400 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:22.417 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:15 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:22.506 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:22.549 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:22.550 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:22.550 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:22.631 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-08-16T02:36:22.981 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:23.054 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:36:23.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:23.190 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:16 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:23.319 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:23.319 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:23.319 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-08-16T02:36:23.319 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 32 k 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 32 k 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 142 k 2024-08-16T02:36:23.320 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:23.496 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.0.0-5812.g6f386a3f.el9.x86_6 180 kB/s | 32 kB 00:00 2024-08-16T02:36:23.497 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:23.497 INFO:teuthology.orchestra.run.smithi152.stdout:Total 179 kB/s | 32 kB 00:00 2024-08-16T02:36:23.497 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:23.501 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:23.501 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:23.514 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:23.514 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:23.596 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:23.709 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:24.058 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:24.232 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:24.281 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:24.281 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:24.282 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:24.282 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:24.282 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:24.282 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:24.468 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-08-16T02:36:24.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo yum -y install rbd-nbd 2024-08-16T02:36:25.017 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:18 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:25.041 INFO:teuthology.orchestra.run.smithi116.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:34:05 AM UTC. 2024-08-16T02:36:25.104 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:25.147 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:25.148 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:25.148 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout:Dependencies resolved. 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout: Package Architecture Version Repository Size 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout:Installing: 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:36:25.174 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction Summary 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:================================================================================ 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Install 1 Package 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Total download size: 177 k 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Installed size: 685 k 2024-08-16T02:36:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Downloading Packages: 2024-08-16T02:36:25.233 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-08-16T02:36:25.406 INFO:teuthology.orchestra.run.smithi116.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 767 kB/s | 177 kB 00:00 2024-08-16T02:36:25.407 INFO:teuthology.orchestra.run.smithi116.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:25.407 INFO:teuthology.orchestra.run.smithi116.stdout:Total 763 kB/s | 177 kB 00:00 2024-08-16T02:36:25.407 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction check 2024-08-16T02:36:25.416 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction check succeeded. 2024-08-16T02:36:25.416 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction test 2024-08-16T02:36:25.480 INFO:teuthology.orchestra.run.smithi116.stdout:Transaction test succeeded. 2024-08-16T02:36:25.480 INFO:teuthology.orchestra.run.smithi116.stdout:Running transaction 2024-08-16T02:36:25.647 INFO:teuthology.orchestra.run.smithi116.stdout: Preparing : 1/1 2024-08-16T02:36:25.737 INFO:teuthology.orchestra.run.smithi116.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:25.780 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:18 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:25.868 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:25.912 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:25.913 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:25.913 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:25.989 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-08-16T02:36:26.123 INFO:teuthology.orchestra.run.smithi116.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout:Installed: 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:26.389 INFO:teuthology.orchestra.run.smithi116.stdout:Complete! 2024-08-16T02:36:26.543 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:19 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:26.619 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:26.630 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:26.672 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:26.672 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:26.673 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:26.750 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-08-16T02:36:27.310 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:20 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:27.396 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:27.438 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:27.439 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:27.440 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:27.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-08-16T02:36:28.074 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:21 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:28.162 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:28.205 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:28.206 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:28.206 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:28.278 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-08-16T02:36:28.838 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:21 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:28.925 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 is already installed. 2024-08-16T02:36:28.968 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:28.969 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-08-16T02:36:28.969 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:29.052 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-08-16T02:36:29.643 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:22 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:29.801 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 91 k 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 91 k 2024-08-16T02:36:29.802 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 321 k 2024-08-16T02:36:29.803 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:30.016 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 427 kB/s | 91 kB 00:00 2024-08-16T02:36:30.016 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:30.017 INFO:teuthology.orchestra.run.smithi152.stdout:Total 424 kB/s | 91 kB 00:00 2024-08-16T02:36:30.017 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:30.026 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:30.026 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:30.092 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:30.093 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:30.239 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:30.340 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:30.785 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:31.032 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:31.271 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-08-16T02:36:31.843 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:24 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:31.975 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 3.5 M 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:31.976 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:31.977 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:31.977 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.5 M 2024-08-16T02:36:31.977 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 30 M 2024-08-16T02:36:31.977 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:32.346 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 9.5 MB/s | 3.5 MB 00:00 2024-08-16T02:36:32.347 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:32.350 INFO:teuthology.orchestra.run.smithi152.stdout:Total 9.4 MB/s | 3.5 MB 00:00 2024-08-16T02:36:32.351 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:32.358 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:32.358 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:32.442 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:32.442 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:33.020 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:33.088 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-16T02:36:33.114 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:34.365 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:34.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-08-16T02:36:35.234 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:28 ago on Fri 16 Aug 2024 02:34:07 AM UTC. 2024-08-16T02:36:35.366 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-5812.g6f386a3f.el9 ceph 177 k 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:35.367 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 177 k 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 685 k 2024-08-16T02:36:35.368 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-08-16T02:36:35.682 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.0.0-5812.g6f386a3f.el9.x86_64.rpm 563 kB/s | 177 kB 00:00 2024-08-16T02:36:35.683 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-08-16T02:36:35.683 INFO:teuthology.orchestra.run.smithi152.stdout:Total 560 kB/s | 177 kB 00:00 2024-08-16T02:36:35.683 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-08-16T02:36:35.692 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-08-16T02:36:35.693 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-08-16T02:36:35.756 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-08-16T02:36:35.756 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-08-16T02:36:35.925 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-08-16T02:36:36.021 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:36.431 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 1/1 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-5812.g6f386a3f.el9.x86_64 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:36.690 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-08-16T02:36:36.944 DEBUG:teuthology.parallel:result is None 2024-08-16T02:36:36.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:36:37.101 DEBUG:teuthology.orchestra.run.smithi116:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:36:37.127 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:36:37.127 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:36:37.128 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:36:37.129 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:36:37.269 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-16T02:36:37.290 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-5812.g6f386a3f.el9 2024-08-16T02:36:37.290 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5812.g6f386a3f.el9 2024-08-16T02:36:37.290 INFO:teuthology.task.install:The correct ceph version 19.0.0-5812.g6f386a3f is installed. 2024-08-16T02:36:37.292 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-16T02:36:37.293 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:37.293 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:36:37.323 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:37.324 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-16T02:36:37.356 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-16T02:36:37.356 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:37.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:36:37.391 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:36:37.459 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:37.459 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-08-16T02:36:37.485 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-16T02:36:37.550 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-16T02:36:37.550 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:37.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:36:37.578 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:36:37.644 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:37.644 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-16T02:36:37.670 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-16T02:36:37.743 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-16T02:36:37.743 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:37.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:36:37.768 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:36:37.833 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:37.834 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-08-16T02:36:37.858 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-16T02:36:37.923 INFO:teuthology.run_tasks:Running task ceph... 2024-08-16T02:36:38.005 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-16T02:36:38.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-08-16T02:36:38.008 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-08-16T02:36:38.033 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-16T02:36:38.033 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:36:38.073 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-16T02:36:38.097 INFO:tasks.ceph:Creating extra log directories... 2024-08-16T02:36:38.097 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:36:38.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-16T02:36:38.170 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-16T02:36:38.171 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-16T02:36:38.171 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419', 'branch': 'wip-yingxin-testing-2024-08-15-1551', 'description': 'crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7857419', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-testing-2024-08-15-1551'}, 'ceph': {'conf': {'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'workunit': {'branch': 'wip-yingxin-testing-2024-08-15-1551', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-testing-2024-08-15-1551', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGgJy+r4YXXHKqjf+iPZ773bSWUgJiAtywUiTZ3yPFMtIyOAaT9jQ0/D5LRRWKk6rYOqLh+6v2TyadFXMDVIN+M=', 'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJN0eLkAaQKXqliGcXiWf+GWaN8aDOqj1a33CqLWfDhU0S0tMZG/K2as32jjpaHd/R4tCg76H4QPgNsz9vK9jY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'client': {'debug ms': 1}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '6f386a3f0b1fe6a4d7219b54bf963d790bc84004', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rados/test.sh --crimson', 'rados/test_pool_quota.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '67db276914d29505c9707d04f3a3b8fb9d8aee4b', 'timestamp': '2024-08-16_01:59:29', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-16T02:36:38.171 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:36:38.203 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-16T02:36:38.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:36:38.259 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-08-16T02:36:38.283 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:38.284 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:36:38.336 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:36:38.336 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-08-16T02:36:38.391 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-16 02:36:23.322788616 +0000 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-16 02:33:02.761382288 +0000 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-16 02:33:02.761382288 +0000 2024-08-16T02:36:38.392 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-16 02:33:02.761382288 +0000 2024-08-16T02:36:38.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:36:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-16T02:36:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-16T02:36:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000198312 s, 2.6 MB/s 2024-08-16T02:36:38.455 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:36:38.511 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-08-16T02:36:38.566 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-16 02:36:23.323788600 +0000 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-16 02:33:03.094376805 +0000 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-16 02:33:03.094376805 +0000 2024-08-16T02:36:38.567 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-16 02:33:03.094376805 +0000 2024-08-16T02:36:38.568 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:36:38.629 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-16T02:36:38.630 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-16T02:36:38.630 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000161115 s, 3.2 MB/s 2024-08-16T02:36:38.631 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:36:38.687 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-08-16T02:36:38.742 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-16 02:36:23.323788600 +0000 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-16 02:33:03.424371372 +0000 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-16 02:33:03.424371372 +0000 2024-08-16T02:36:38.743 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-16 02:33:03.424371372 +0000 2024-08-16T02:36:38.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:36:38.806 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-16T02:36:38.806 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-16T02:36:38.806 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000225703 s, 2.3 MB/s 2024-08-16T02:36:38.807 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:36:38.864 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-08-16 02:36:23.323788600 +0000 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-08-16 02:33:03.754365939 +0000 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-08-16 02:33:03.754365939 +0000 2024-08-16T02:36:38.919 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: 2024-08-16 02:33:03.754365939 +0000 2024-08-16T02:36:38.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:36:38.982 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-08-16T02:36:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-08-16T02:36:38.983 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00018664 s, 2.7 MB/s 2024-08-16T02:36:38.983 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:36:39.039 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:39.039 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:39.039 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-08-16T02:36:39.058 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-16T02:36:39.058 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-08-16T02:36:39.116 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-16T02:36:39.116 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-08-16 02:36:33.359726592 +0000 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-08-16 02:33:03.738953744 +0000 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-08-16 02:33:03.738953744 +0000 2024-08-16T02:36:39.117 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-08-16 02:33:03.738953744 +0000 2024-08-16T02:36:39.117 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-16T02:36:39.183 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-08-16T02:36:39.183 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-08-16T02:36:39.183 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.00019618 s, 2.6 MB/s 2024-08-16T02:36:39.184 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-16T02:36:39.244 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-08-16T02:36:39.301 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-08-16 02:36:33.360726580 +0000 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-08-16 02:33:04.076949845 +0000 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-08-16 02:33:04.076949845 +0000 2024-08-16T02:36:39.302 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-08-16 02:33:04.076949845 +0000 2024-08-16T02:36:39.302 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-16T02:36:39.366 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-08-16T02:36:39.366 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-08-16T02:36:39.366 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000196517 s, 2.6 MB/s 2024-08-16T02:36:39.367 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-16T02:36:39.425 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-08-16 02:36:33.360726580 +0000 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-08-16 02:33:04.405946050 +0000 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-08-16 02:33:04.405946050 +0000 2024-08-16T02:36:39.482 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-08-16 02:33:04.405946050 +0000 2024-08-16T02:36:39.483 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-16T02:36:39.547 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-08-16T02:36:39.547 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-08-16T02:36:39.547 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000181554 s, 2.8 MB/s 2024-08-16T02:36:39.548 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-16T02:36:39.605 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-08-16 02:36:33.360726580 +0000 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-08-16 02:33:04.735942244 +0000 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-08-16 02:33:04.735942244 +0000 2024-08-16T02:36:39.661 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-08-16 02:33:04.735942244 +0000 2024-08-16T02:36:39.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-16T02:36:39.725 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-08-16T02:36:39.725 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-08-16T02:36:39.725 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000193013 s, 2.7 MB/s 2024-08-16T02:36:39.726 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-16T02:36:39.782 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:39.783 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-16T02:36:39.783 INFO:tasks.ceph:Generating config... 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-16T02:36:39.784 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-16T02:36:39.785 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-16T02:36:39.785 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-16T02:36:39.785 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-16T02:36:39.785 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-08-16T02:36:39.785 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-16T02:36:39.786 INFO:tasks.ceph:[client] debug ms = 1 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms client mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[global] ms service mode = crc 2024-08-16T02:36:39.786 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-16T02:36:39.786 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-16T02:36:39.786 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-08-16T02:36:39.786 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-08-16T02:36:39.786 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-08-16T02:36:39.787 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-08-16T02:36:39.787 INFO:tasks.ceph:Setting up mon.a... 2024-08-16T02:36:39.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-16T02:36:39.862 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2024-08-16T02:36:39.865 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-16T02:36:39.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:39.941 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116')] 2024-08-16T02:36:39.942 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'mon warn on pool no app': False, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug ms': 1}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-08-16T02:36:39.943 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:39.943 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:36:39.997 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:40.088+0000 7fbeac2e1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:40.088+0000 7fbeac2e1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:40.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:40.088+0000 7fbeac2e1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 8b1d1868-3d02-4f61-802a-791ead3a36d9 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 8b1d1868-3d02-4f61-802a-791ead3a36d9 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2024-08-16T02:36:40.090032+0000 2024-08-16T02:36:40.090 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-08-16T02:36:40.090032+0000 2024-08-16T02:36:40.091 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 17 (quincy) 2024-08-16T02:36:40.091 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2024-08-16T02:36:40.091 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-08-16T02:36:40.091 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-08-16T02:36:40.092 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-16T02:36:40.106 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8b1d1868-3d02-4f61-802a-791ead3a36d9... 2024-08-16T02:36:40.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:36:40.149 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-16T02:36:40.189 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-08-16T02:36:40.189 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-08-16T02:36:40.190 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:36:40.191 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-08-16T02:36:40.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = 172.21.15.116 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms client mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms cluster mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms mon client mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms mon cluster mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms mon service mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: ms service mode = crc 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 8b1d1868-3d02-4f61-802a-791ead3a36d9 2024-08-16T02:36:40.193 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-08-16T02:36:40.194 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.195 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: debug monc = 20 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: crimson osd obc lru size = 10 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd blocked scrub grace period = 3600 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd class default list = * 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd class load list = * 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = seastore 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: osd shutdown pgref assert = True 2024-08-16T02:36:40.196 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-08-16T02:36:40.197 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:36:40.198 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon min osdmap epochs = 50 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune interval = 2 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune min = 15 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: paxos service trim min = 10 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on pool no app = False 2024-08-16T02:36:40.199 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default crimson = True 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-08-16T02:36:40.200 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-08-16T02:36:40.201 INFO:teuthology.orchestra.run.smithi116.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = cephx 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-08-16T02:36:40.202 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-08-16T02:36:40.203 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.204 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = 172.21.15.116 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms client mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms cluster mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon client mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon cluster mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon service mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: ms service mode = crc 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = 8b1d1868-3d02-4f61-802a-791ead3a36d9 2024-08-16T02:36:40.205 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-08-16T02:36:40.206 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-08-16T02:36:40.207 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: crimson alien thread cpu cores = 6-7 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: crimson osd obc lru size = 10 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 20 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: osd blocked scrub grace period = 3600 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: osd class default list = * 2024-08-16T02:36:40.208 INFO:teuthology.orchestra.run.smithi152.stdout: osd class load list = * 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = seastore 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: osd shutdown pgref assert = True 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-08-16T02:36:40.209 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 20 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-08-16T02:36:40.210 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon min osdmap epochs = 50 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune interval = 2 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune min = 15 2024-08-16T02:36:40.211 INFO:teuthology.orchestra.run.smithi152.stdout: mon osdmap full prune txsize = 2 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: paxos service trim min = 10 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on pool no app = False 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default crimson = True 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-16T02:36:40.212 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 0-2 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-08-16T02:36:40.213 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 3-5 2024-08-16T02:36:40.214 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-16T02:36:40.215 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-16T02:36:40.319 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-16T02:36:40.320 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:40.320 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-16T02:36:40.336 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:40.336 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-16T02:36:40.391 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:36:40.392 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:40.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:36:40.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:40.466 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:40.466 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:40.522 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:36:40.522 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:40.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-16T02:36:40.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-16T02:36:40.558 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:40.558 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:40.614 INFO:tasks.ceph:Setting up mon nodes... 2024-08-16T02:36:40.614 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-16T02:36:40.614 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:36:40.737 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-16T02:36:40.740 INFO:tasks.ceph:Setting up mds nodes... 2024-08-16T02:36:40.740 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-16T02:36:40.740 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-16T02:36:40.788 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-16T02:36:40.801 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-16T02:36:40.850 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-16T02:36:40.864 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-16T02:36:40.864 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-08-16T02:36:40.864 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:40.889 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:40.889 INFO:tasks.ceph:role: osd.0 2024-08-16T02:36:40.889 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:36:40.889 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:36:40.960 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:40.960 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:40.960 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:40.960 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:40.961 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:41.482 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-08-16T02:36:41.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-08-16T02:36:41.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:41.595 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:41.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:41.695 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:41.695 INFO:tasks.ceph:role: osd.1 2024-08-16T02:36:41.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:36:41.695 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:36:41.768 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:41.769 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:42.233 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-08-16T02:36:42.236 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-08-16T02:36:42.236 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:42.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:42.358 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:42.670 INFO:teuthology.orchestra.run.smithi116.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:42.671 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:42,671 seastar - Reactor backend: linux-aio 2024-08-16T02:36:42.686 INFO:teuthology.orchestra.run.smithi116.stderr:==29472==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:42,797 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:42.797 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:42,797 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-16T02:36:42.798 INFO:teuthology.orchestra.run.smithi116.stderr:DEBUG 2024-08-16 02:36:42,798 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:Suppressions used: 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr: count bytes template 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:43.376 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-16T02:36:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:43.435 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:43,435 seastar - Reactor backend: linux-aio 2024-08-16T02:36:43.444 INFO:teuthology.orchestra.run.smithi116.stderr:==29470==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbac99a9 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54dc6ba 0xbafcc73 0xbb07179 0xbb07345 0xbaaf513 0xbaafa42 0xbab5a05 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.712 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3afc6c7 0x3afc9ba 0xbae2576 0xbb1017b 0xbb107c5 0xbaae50c 0x981cebf 0x9ae5c54 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x923b1fc 0x923b718 0x923b8a1 0x923bae8 0x923bdc0 0x923be75 0x9556f6e 0x95571d3 0x958fe43 0x96a9a42 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x981e50b 0xb9295f4 0xbaadc57 0x981cebf 0x9ae623a 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x923b1fc 0x923b718 0x923b8a1 0x923bae8 0x923bdc0 0x923be75 0x9556f6e 0x95571d3 0x958fe43 0x96a9a42 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.859 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x8b4ff 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:43.882 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d883b 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:43.940 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.972 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362f6 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.972 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:43.972 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82e9 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:43.973 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.030 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8930 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:44.030 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.030 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x2650f 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:44.030 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.311 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xb930898 0xb9309b6 0xb930c36 0xb930cef 0xb9311e1 0xb9315ec 0xb9318ab 0xb932368 0xbb021f2 0xbaa819b 0x9ae6277 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:44.312 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.312 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36340 0xbb06fdd 0xbb071d8 0xbb07345 0xbaaf549 0xbaafa42 0xbab5a05 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:44.312 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.394 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7c2a 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:44.394 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:44.451 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362fa 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:44.452 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:Suppressions used: 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr: count bytes template 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:45.312 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-16T02:36:45.338 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-16T02:36:45.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:45.664 INFO:teuthology.orchestra.run.smithi116.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:45.665 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:45,665 seastar - Reactor backend: linux-aio 2024-08-16T02:36:45.670 INFO:teuthology.orchestra.run.smithi116.stderr:==29520==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:45,779 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:45.779 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:45,779 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-16T02:36:45.780 INFO:teuthology.orchestra.run.smithi116.stderr:DEBUG 2024-08-16 02:36:45,780 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:Suppressions used: 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr: count bytes template 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:46.312 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-16T02:36:46.369 INFO:teuthology.orchestra.run.smithi116.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:46.373 INFO:teuthology.orchestra.run.smithi116.stderr:INFO 2024-08-16 02:36:46,373 seastar - Reactor backend: linux-aio 2024-08-16T02:36:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:==29518==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:46.649 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xef7e7 0x4a0a5 0x35b7c 0x3689e 0x36e51c8 0x36e539f 0x5c57b5e 0xbab6e5d 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.650 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.650 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36889 0x3afc839 0x3afc9ba 0xbadb0c0 0xbb1555f 0xbb158ea 0xbab6dd6 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.650 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.836 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x155fb 0x4a09d 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:46.836 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.893 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362a7 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:46.894 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x362db 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.927 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a065 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bda 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113c69 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9558d09 0x989744c 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:47.001 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x161cdf 0xc9c6c 0xca7a6 0x35920f0 0x3afc7c2 0x3afc9ba 0xbb10a6a 0xbaae50c 0x981cebf 0x9ae623a 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x368cd 0x3afc839 0x3afc9ba 0xbac630f 0xbac75e2 0xbac785c 0xbac7a04 0xbac97bf 0xbac99bb 0xbb0dc39 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.263 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.345 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8b4c 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:47.346 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113bbf 0x4a0bd 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:47.409 INFO:teuthology.orchestra.run.smithi116.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xe49aa 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:47.409 INFO:teuthology.orchestra.run.smithi116.stderr:kernel callstack: 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:Suppressions used: 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr: count bytes template 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr:----------------------------------------------------- 2024-08-16T02:36:48.322 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-16T02:36:48.350 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-16T02:36:48.374 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:48.405 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:48.405 INFO:tasks.ceph:role: osd.2 2024-08-16T02:36:48.405 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:36:48.405 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-16T02:36:48.481 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:48.481 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:48.481 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:48.482 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:48.979 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-08-16T02:36:48.981 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-08-16T02:36:48.981 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:49.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:49.145 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:49.212 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-08-16T02:36:49.212 INFO:tasks.ceph:role: osd.3 2024-08-16T02:36:49.212 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:36:49.213 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-16T02:36:49.286 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-16T02:36:49.287 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-16T02:36:49.745 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-08-16T02:36:49.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-08-16T02:36:49.747 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:49.836 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:49.865 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:50.195 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:50.196 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:50,196 seastar - Reactor backend: linux-aio 2024-08-16T02:36:50.212 INFO:teuthology.orchestra.run.smithi152.stderr:==29065==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:50.334 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:50,334 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:50.334 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:50,334 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 0-2 2024-08-16T02:36:50.334 INFO:teuthology.orchestra.run.smithi152.stderr:DEBUG 2024-08-16 02:36:50,335 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:50.920 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:50.920 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-08-16T02:36:50.920 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-08-16T02:36:50.921 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:50.921 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:50.921 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-08-16T02:36:50.973 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:50.976 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:50,977 seastar - Reactor backend: linux-aio 2024-08-16T02:36:50.986 INFO:teuthology.orchestra.run.smithi152.stderr:==29061==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:51.257 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a0b5 0x35b7c 0x36eee 0xbac74a3 0xbac785c 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.257 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x5c5745b 0x5c5764c 0x5c57903 0x5c579f8 0xbaf968a 0xbb04840 0xbb04a5b 0xbb04b0b 0xbab55cb 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.258 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.258 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.315 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x3596297 0x3621816 0x3667013 0x3667bc6 0x9ed4f8e 0x9ed516e 0xc5aa067 0xc5ab201 0x9e54efe 0x9e2ca49 0x9e39b03 0x99e55e7 0x99e5bed 0x98decca 0x92f408f 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.315 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x36911 0x60ab19f 0x60ab323 0x60ab5ab 0xbb10c4b 0xbb10d7c 0xbb10f68 0xbb11104 0xbb1149a 0xbb116ac 0xbaaf61d 0xbaafa42 0xbab74ad 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9e29a36 0x9e39d7e 0x99e55e7 0x99e5bed 0x98decca 0x92f408f 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.315 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.315 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.458 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d888b 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:51.458 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.515 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d89eb 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:51.516 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.531 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8b54 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.531 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.550 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7d4c 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.550 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.588 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7b49 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.589 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.607 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xef7f8 0x4a0a5 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.608 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.884 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xc6258 0x5c286bd 0xbac7552 0xbac785c 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.884 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.884 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x4a0c5 0x35b7c 0x3629e 0x5befff5 0x8fa0ee3 0xbb072f0 0xbaaf549 0xbaafa42 0xbab5a05 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:51.884 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:51.991 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x15603 0x4a09d 0x35b7c 0x3629e 0x54d8a2d 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:51.992 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:52.049 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d83fe 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:52.049 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:52.908 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:52.908 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-08-16T02:36:52.908 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-08-16T02:36:52.909 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:52.909 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:52.909 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-08-16T02:36:52.934 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-16T02:36:52.960 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:53.260 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:53.261 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:53,261 seastar - Reactor backend: linux-aio 2024-08-16T02:36:53.266 INFO:teuthology.orchestra.run.smithi152.stderr:==29113==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:53.380 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:53,379 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-08-16T02:36:53.380 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:53,379 [shard 0:main] osd - get_early_config: set --thread-affinity 1 --cpuset 3-5 2024-08-16T02:36:53.380 INFO:teuthology.orchestra.run.smithi152.stderr:DEBUG 2024-08-16 02:36:53,380 [shard 0:main] seastar - reactor::drain 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:53.929 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-08-16T02:36:53.984 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-08-16T02:36:53.988 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-08-16 02:36:53,988 seastar - Reactor backend: linux-aio 2024-08-16T02:36:53.997 INFO:teuthology.orchestra.run.smithi152.stderr:==29111==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-08-16T02:36:54.226 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x105ab7 0x2a517 0xe0e5c 0x5c57472 0x5c5764c 0x5c57903 0x5c578c8 0x5c579f8 0xbaf968a 0xbab9efc 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x967d442 0x967d7de 0x967d963 0x967da05 0x967db1e 0xbf2c63b 0xbf2c842 0xbf2c918 0xbf2ca53 0xbf2ce47 0xbf2c20a 0xbf2c2ea 0xbf2c3fe 0xbf2c0c1 0x94c7872 0x95add6e 0x92ea331 0x92ea50b 0x92ea646 0x92ea700 0x92ea7da 0x92ead26 0x92eaffe 0x92eb0b3 0x9561672 0x95618d7 0x958fd08 0x96a8aca 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:54.226 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.272 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbb10483 0xbaae50c 0x981cebf 0x9ae623a 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.272 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.272 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x113ca8 0x4a0bd 0x35b7c 0x36eee 0xbac74a3 0xbac785c 0xbac7a04 0xbac97bf 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92f3e8b 0x92f4e2a 0x92f4ead 0x92f4f32 0x92f5034 0x92f512c 0x92f51b2 0x92f526b 0x92f5421 0x92f5559 0x92f55d4 0x92f5659 0x92f57b4 0x92f58ce 0x92f5953 0x92f5a31 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.272 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.492 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d886a 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:54.493 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.550 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x9e4b334 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:54.551 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.577 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7b47 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.577 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.577 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7d10 0x9e4b359 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.577 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.635 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8819 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.635 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.635 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8996 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.635 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.889 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0xbac981a 0xbac9c0a 0xbab5fba 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.889 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.889 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x2d560 0x2d86b 0x29edb 0xdfdb3 0xb93070d 0xb9307c0 0xb9308d6 0xb9309b6 0xb930c36 0xb9311e1 0xb9315ec 0xb9318ab 0xbb0cd4c 0xbb0cfe8 0xbb0d1af 0xbb0d423 0xbb0d52c 0xbb0d6a8 0xbb0de54 0xbab61fd 0xbab9c26 0xbaa4bc4 0xbaa4fa8 0x9ae6305 0x9afde99 0x9997a62 0x992b2c9 0x923aae7 0x92d82dc 0x92d8d80 0x92d8e03 0x92d8e88 0x92d8f8a 0x92d9082 0x92d9108 0x92d91c1 0x92d9377 0x92d94af 0x92d952a 0x92d95af 0x92d970a 0x92d9824 0x92d98a9 0x92d9987 0x9523738 0x95ddc3f 0x95dddbc 0x96b69f5 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:54.889 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:54.972 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d8910 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:54.972 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:55.030 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7f20 0x9e4b36e 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd3b967 0xba0aef9 0xba0b875 0x37892c2 0x2958f 0x2963f 0x3577994 2024-08-16T02:36:55.030 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:55.036 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d82e9 0x9e4b37b 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:55.036 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:55.037 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x6bddd 0xbd88bdb 0xbc5ff2f 0xbc6149a 0xbc616b0 0xbc61806 0xbc61ccf 0x3e6ef 0x54d7a23 0x54d89ce 0x9e4b340 0x9ebc593 0x9efd58b 0x9e525ec 0x9941839 0x986197a 0x9885eb8 0x9886017 0x9897434 0x9897604 0x9897912 0x9897c21 0x9897d80 0x9897dfb 0x9898042 0x9898306 0x9898396 0x989843a 0x95883c2 0x9588627 0x9590bcc 0x969888a 0xbc72bb5 0xbc8c47d 0xbd3a319 0xbd4cec8 0xbd4d4af 0x7acbf63 0xc6f5196 0x89c01 0x10ec3f 2024-08-16T02:36:55.037 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-08-16T02:36:55.926 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:55.927 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-08-16T02:36:55.927 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-08-16T02:36:55.927 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-08-16T02:36:55.927 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-08-16T02:36:55.927 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-08-16T02:36:55.954 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-16T02:36:55.979 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-16T02:36:55.979 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:55.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-16T02:36:56.009 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:56.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-16T02:36:56.076 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:56.076 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-16T02:36:56.101 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:56.101 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-16T02:36:56.166 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:56.166 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-16T02:36:56.231 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:56.231 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-16T02:36:56.248 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:56.249 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-16T02:36:56.287 INFO:tasks.ceph:Adding keys to all mons... 2024-08-16T02:36:56.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-16T02:36:56.315 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-08-16T02:36:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA7u75m3nN1IhAAq7BBAYHCd4UpbggetpuFAg== 2024-08-16T02:36:56.315 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-08-16T02:36:56.315 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA+u75miNvZHhAASzgUB5lvr/Ze6kNM+zfktQ== 2024-08-16T02:36:56.315 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4u75m0Dv3KxAAHmvnIFIuxW8b8JeZCQ5KxQ== 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBDu75mHkJPBxAAM6Ch88xuTLz0/77SL0/KBQ== 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQBGu75mT6EaCBAA9nDhyVS8Qg3CjiupxBJvLw== 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4u75mpIQHLxAAuQnkxamSmspS7ZQSCY5rIA== 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-08-16T02:36:56.316 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQA4u75m7yG2MhAA2vbDU6f4enKo3XSKRWNlIQ== 2024-08-16T02:36:56.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:56.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:56.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-16T02:36:56.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:56.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-16T02:36:56.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:36:56.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-16T02:36:56.792 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-16T02:36:56.793 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-16T02:36:56.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-16T02:36:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:56.926+0000 7f0075f2cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:56.926+0000 7f0075f2cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:56.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-16T02:36:56.964 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-16T02:36:57.020 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-16T02:36:57.020 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-16T02:36:57.020 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-08-16T02:36:57.062 INFO:tasks.ceph.mon.a:Started 2024-08-16T02:36:57.062 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-16T02:36:57.062 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-16T02:36:57.063 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-08-16T02:36:57.065 INFO:tasks.ceph.mgr.x:Started 2024-08-16T02:36:57.065 DEBUG:tasks.ceph:set 0 configs 2024-08-16T02:36:57.065 INFO:tasks.ceph:Setting crush tunables to default 2024-08-16T02:36:57.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2024-08-16T02:36:57.142 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-08-16T02:36:57.141+0000 7fda13636c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.143 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-08-16T02:36:57.141+0000 7fda13636c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.143 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-08-16T02:36:57.142+0000 7fda13636c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.172 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-08-16T02:36:57.171+0000 7f2411b97000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.167+0000 7f17bd613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.174+0000 7f17bd613640 1 Processor -- start 2024-08-16T02:36:57.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.174+0000 7f17bd613640 1 -- start start 2024-08-16T02:36:57.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17bd613640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17b81332f0 0x7f17b8133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:57.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17bd613640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17b80a0a10 con 0x7f17b806e8b0 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17bd613640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f17b809e9e0 con 0x7f17b81332f0 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17b6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17b81332f0 0x7f17b8133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17b6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17b81332f0 0x7f17b8133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60274/0 (socket says 172.21.15.116:60274) 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.175+0000 7f17b6ffd640 1 -- 172.21.15.116:0/2179152560 learned_addr learned my addr 172.21.15.116:0/2179152560 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.176+0000 7f17b6ffd640 1 -- 172.21.15.116:0/2179152560 >> v1:172.21.15.116:6789/0 conn(0x7f17b806e8b0 legacy=0x7f17b806ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.176+0000 7f17b6ffd640 1 -- 172.21.15.116:0/2179152560 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b8070a40 con 0x7f17b81332f0 2024-08-16T02:36:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.177+0000 7f17b6ffd640 1 --2- 172.21.15.116:0/2179152560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b81332f0 0x7f17b8133700 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385b50ed84cfaa42 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.177+0000 7f17b5ffb640 1 -- 172.21.15.116:0/2179152560 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f17ac009e50 con 0x7f17b81332f0 2024-08-16T02:36:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.177+0000 7f17b5ffb640 1 -- 172.21.15.116:0/2179152560 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f17ac00e230 con 0x7f17b81332f0 2024-08-16T02:36:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.177+0000 7f17b5ffb640 1 -- 172.21.15.116:0/2179152560 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f17ac00e4d0 con 0x7f17b81332f0 2024-08-16T02:36:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 -- 172.21.15.116:0/2179152560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b81332f0 msgr2=0x7f17b8133700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 --2- 172.21.15.116:0/2179152560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b81332f0 0x7f17b8133700 crc :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 -- 172.21.15.116:0/2179152560 shutdown_connections 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 --2- 172.21.15.116:0/2179152560 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b81332f0 0x7f17b8133700 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 -- 172.21.15.116:0/2179152560 >> 172.21.15.116:0/2179152560 conn(0x7f17b806caf0 msgr2=0x7f17b806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 -- 172.21.15.116:0/2179152560 shutdown_connections 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 1 -- 172.21.15.116:0/2179152560 wait complete. 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.178+0000 7f17bd613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.180+0000 7f17bd613640 1 Processor -- start 2024-08-16T02:36:57.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.180+0000 7f17bd613640 1 -- start start 2024-08-16T02:36:57.181 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-08-16T02:36:57.180+0000 7f2411b97000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.181 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-08-16T02:36:57.181+0000 7f2411b97000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17bd613640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17bd613640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17b8073530 con 0x7f17b806e8b0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60276/0 (socket says 172.21.15.116:60276) 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17b6ffd640 1 -- 172.21.15.116:0/1883024576 learned_addr learned my addr 172.21.15.116:0/1883024576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.181+0000 7f17b6ffd640 1 -- 172.21.15.116:0/1883024576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b8125220 con 0x7f17b806e8b0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f17b6ffd640 1 --2- 172.21.15.116:0/1883024576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f17ac00ee20 con 0x7f17b806e8b0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f17ac015420 con 0x7f17b806e8b0 2024-08-16T02:36:57.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b8123a40 con 0x7f17b806e8b0 2024-08-16T02:36:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f17ac0156c0 con 0x7f17b806e8b0 2024-08-16T02:36:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b8124f10 con 0x7f17b806e8b0 2024-08-16T02:36:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f17ac015c00 con 0x7f17b806e8b0 2024-08-16T02:36:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.182+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (crc 0 0 0) 0x7f17ac011380 con 0x7f17b806e8b0 2024-08-16T02:36:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.183+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b81260a0 con 0x7f17b806e8b0 2024-08-16T02:36:57.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.185+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f17b81260a0 con 0x7f17b806e8b0 2024-08-16T02:36:57.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.249+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f17b8076690 con 0x7f17b806e8b0 2024-08-16T02:36:57.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.250+0000 7f179f7fe640 1 -- 172.21.15.116:0/1883024576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (crc 0 0 0) 0x7f17ac014e50 con 0x7f17b806e8b0 2024-08-16T02:36:57.251 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 msgr2=0x7f17b81220f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 --2- 172.21.15.116:0/1883024576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 shutdown_connections 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 --2- 172.21.15.116:0/1883024576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17b806e8b0 0x7f17b81220f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 >> 172.21.15.116:0/1883024576 conn(0x7f17b806caf0 msgr2=0x7f17b8073690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 shutdown_connections 2024-08-16T02:36:57.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.251+0000 7f17bd613640 1 -- 172.21.15.116:0/1883024576 wait complete. 2024-08-16T02:36:57.263 INFO:tasks.ceph:check_enable_crimson: True 2024-08-16T02:36:57.263 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-08-16T02:36:57.263 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-08-16T02:36:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.363+0000 7f2c96112640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.369+0000 7f2c96112640 1 Processor -- start 2024-08-16T02:36:57.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.369+0000 7f2c96112640 1 -- start start 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c96112640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2c90135490 0x7f2c901358a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c96112640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c900a4f10 con 0x7f2c90071390 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c96112640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2c900a5140 con 0x7f2c90135490 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c8f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2c90135490 0x7f2c901358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c8effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2c90071390 0x7f2c900717a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44384/0 (socket says 172.21.15.116:44384) 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.370+0000 7f2c8effd640 1 -- 172.21.15.116:0/153331248 learned_addr learned my addr 172.21.15.116:0/153331248 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8f7fe640 1 -- 172.21.15.116:0/153331248 >> v1:172.21.15.116:6789/0 conn(0x7f2c90071390 legacy=0x7f2c900717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8f7fe640 1 -- 172.21.15.116:0/153331248 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c90072b20 con 0x7f2c90135490 2024-08-16T02:36:57.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8f7fe640 1 --2- 172.21.15.116:0/153331248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90135490 0x7f2c901358a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6416a8f187021938 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8e7fc640 1 -- 172.21.15.116:0/153331248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c80009d90 con 0x7f2c90135490 2024-08-16T02:36:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8e7fc640 1 -- 172.21.15.116:0/153331248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2c8000e230 con 0x7f2c90135490 2024-08-16T02:36:57.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.371+0000 7f2c8e7fc640 1 -- 172.21.15.116:0/153331248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c8000e4d0 con 0x7f2c90135490 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 -- 172.21.15.116:0/153331248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90135490 msgr2=0x7f2c901358a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 --2- 172.21.15.116:0/153331248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90135490 0x7f2c901358a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 -- 172.21.15.116:0/153331248 shutdown_connections 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 --2- 172.21.15.116:0/153331248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90135490 0x7f2c901358a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 -- 172.21.15.116:0/153331248 >> 172.21.15.116:0/153331248 conn(0x7f2c900a5880 msgr2=0x7f2c900a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 -- 172.21.15.116:0/153331248 shutdown_connections 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 1 -- 172.21.15.116:0/153331248 wait complete. 2024-08-16T02:36:57.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.372+0000 7f2c96112640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:57.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.373+0000 7f2c96112640 1 Processor -- start 2024-08-16T02:36:57.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.373+0000 7f2c96112640 1 -- start start 2024-08-16T02:36:57.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.374+0000 7f2c96112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:57.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.374+0000 7f2c96112640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2c901267d0 con 0x7f2c90071390 2024-08-16T02:36:57.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.374+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:57.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60280/0 (socket says 172.21.15.116:60280) 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c8f7fe640 1 -- 172.21.15.116:0/1151901437 learned_addr learned my addr 172.21.15.116:0/1151901437 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c8f7fe640 1 -- 172.21.15.116:0/1151901437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c90128510 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c8f7fe640 1 --2- 172.21.15.116:0/1151901437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c80014620 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2c80014780 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c90127880 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2c80014a20 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c90125f40 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f2c80013400 con 0x7f2c90071390 2024-08-16T02:36:57.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.375+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (crc 0 0 0) 0x7f2c80019040 con 0x7f2c90071390 2024-08-16T02:36:57.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.376+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c900008d0 con 0x7f2c90071390 2024-08-16T02:36:57.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.378+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f2c80013890 con 0x7f2c90071390 2024-08-16T02:36:57.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:57.454+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true} v 0) -- 0x7f2c90127540 con 0x7f2c90071390 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:2024-08-16T02:36:57.454+0000 7fda09ff4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:Please be aware that this feature is experimental, untested, 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:unsupported, and may result in data corruption, data loss, 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:and/or irreparable damage to your cluster. Do not use 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr:feature with important data. 2024-08-16T02:36:57.455 INFO:tasks.ceph.mon.a.smithi116.stderr: 2024-08-16T02:36:58.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.251+0000 7f2c77fff640 1 -- 172.21.15.116:0/1151901437 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd set-allow-crimson", "yes_i_really_mean_it": true}]=0 v3) ==== 99+0+0 (crc 0 0 0) 0x7f2c800059f0 con 0x7f2c90071390 2024-08-16T02:36:58.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.252+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 msgr2=0x7f2c901244c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.252+0000 7f2c96112640 1 --2- 172.21.15.116:0/1151901437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.253+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 shutdown_connections 2024-08-16T02:36:58.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.253+0000 7f2c96112640 1 --2- 172.21.15.116:0/1151901437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2c90071390 0x7f2c901244c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.253+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 >> 172.21.15.116:0/1151901437 conn(0x7f2c900a5880 msgr2=0x7f2c9006fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:58.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.253+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 shutdown_connections 2024-08-16T02:36:58.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:58.253+0000 7f2c96112640 1 -- 172.21.15.116:0/1151901437 wait complete. 2024-08-16T02:36:58.267 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-16T02:36:58.267 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:58.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-16T02:36:58.296 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:36:58.296 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-16T02:36:58.362 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:58.362 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-16T02:36:58.388 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-08-16T02:36:58.388 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-16T02:36:58.454 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 18f869dd-54a7-4f0a-bd69-be3b5fd67da7 0 2024-08-16T02:36:58.602 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.596+0000 7efe71e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:58.609 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.602+0000 7efe71e57640 1 Processor -- start 2024-08-16T02:36:58.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe71e57640 1 -- start start 2024-08-16T02:36:58.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe71e57640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe6c111890 0x7efe6c11d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:58.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe71e57640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe6c11d640 con 0x7efe6c110d70 2024-08-16T02:36:58.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe71e57640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efe6c11e7e0 con 0x7efe6c111890 2024-08-16T02:36:58.610 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe63fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe6c111890 0x7efe6c11d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe63fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe6c111890 0x7efe6c11d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45828/0 (socket says 172.21.15.152:45828) 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.603+0000 7efe63fff640 1 -- 172.21.15.152:0/1381902163 learned_addr learned my addr 172.21.15.152:0/1381902163 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe63fff640 1 -- 172.21.15.152:0/1381902163 >> v1:172.21.15.116:6789/0 conn(0x7efe6c110d70 legacy=0x7efe6c111180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe63fff640 1 -- 172.21.15.152:0/1381902163 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c11e920 con 0x7efe6c111890 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe63fff640 1 --2- 172.21.15.152:0/1381902163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c111890 0x7efe6c11d100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3444758b23f0d417 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe6affd640 1 -- 172.21.15.152:0/1381902163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efe54009e50 con 0x7efe6c111890 2024-08-16T02:36:58.611 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe6affd640 1 -- 172.21.15.152:0/1381902163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efe5400e230 con 0x7efe6c111890 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.604+0000 7efe6affd640 1 -- 172.21.15.152:0/1381902163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efe5400e4d0 con 0x7efe6c111890 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 -- 172.21.15.152:0/1381902163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c111890 msgr2=0x7efe6c11d100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 --2- 172.21.15.152:0/1381902163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c111890 0x7efe6c11d100 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 -- 172.21.15.152:0/1381902163 shutdown_connections 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 --2- 172.21.15.152:0/1381902163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c111890 0x7efe6c11d100 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 -- 172.21.15.152:0/1381902163 >> 172.21.15.152:0/1381902163 conn(0x7efe6c070720 msgr2=0x7efe6c072b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:58.612 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 -- 172.21.15.152:0/1381902163 shutdown_connections 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 1 -- 172.21.15.152:0/1381902163 wait complete. 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.605+0000 7efe71e57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.606+0000 7efe71e57640 1 Processor -- start 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.606+0000 7efe71e57640 1 -- start start 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.606+0000 7efe71e57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.606+0000 7efe71e57640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe6c1a70e0 con 0x7efe6c110d70 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.607+0000 7efe6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:58.613 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.607+0000 7efe6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45832/0 (socket says 172.21.15.152:45832) 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.607+0000 7efe6b7fe640 1 -- 172.21.15.152:0/3445460738 learned_addr learned my addr 172.21.15.152:0/3445460738 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.607+0000 7efe6b7fe640 1 -- 172.21.15.152:0/3445460738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe6c1a7220 con 0x7efe6c110d70 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.607+0000 7efe6b7fe640 1 --2- 172.21.15.152:0/3445460738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efe5c00c950 con 0x7efe6c110d70 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7efe5c00cab0 con 0x7efe6c110d70 2024-08-16T02:36:58.614 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe6c1a74d0 con 0x7efe6c110d70 2024-08-16T02:36:58.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7efe5c00cd50 con 0x7efe6c110d70 2024-08-16T02:36:58.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe6c1ac3f0 con 0x7efe6c110d70 2024-08-16T02:36:58.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7efe5c0116a0 con 0x7efe6c110d70 2024-08-16T02:36:58.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.608+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 747+0+0 (crc 0 0 0) 0x7efe5c010dc0 con 0x7efe6c110d70 2024-08-16T02:36:58.615 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.609+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe6c11e240 con 0x7efe6c110d70 2024-08-16T02:36:58.616 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.611+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7efe5c011b30 con 0x7efe6c110d70 2024-08-16T02:36:58.687 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.687+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "18f869dd-54a7-4f0a-bd69-be3b5fd67da7", "id": 0} v 0) -- 0x7efe6c111890 con 0x7efe6c110d70 2024-08-16T02:36:58.689 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.688+0000 7efe68ff9640 1 -- 172.21.15.152:0/3445460738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18f869dd-54a7-4f0a-bd69-be3b5fd67da7", "id": 0}]=0 v4) ==== 112+0+2 (crc 0 0 0) 0x7efe5c011da0 con 0x7efe6c110d70 2024-08-16T02:36:58.689 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-08-16T02:36:58.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 msgr2=0x7efe6c1a6b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 --2- 172.21.15.152:0/3445460738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 shutdown_connections 2024-08-16T02:36:58.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 --2- 172.21.15.152:0/3445460738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe6c110d70 0x7efe6c1a6b10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.690 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 >> 172.21.15.152:0/3445460738 conn(0x7efe6c070720 msgr2=0x7efe6c116f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:58.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 shutdown_connections 2024-08-16T02:36:58.691 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.689+0000 7efe71e57640 1 -- 172.21.15.152:0/3445460738 wait complete. 2024-08-16T02:36:58.701 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new d5599bff-f9e3-4704-a9c2-f90504977470 1 2024-08-16T02:36:58.810 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.804+0000 7ff0ca992640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:58.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.810+0000 7ff0ca992640 1 Processor -- start 2024-08-16T02:36:58.811 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.810+0000 7ff0ca992640 1 -- start start 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.810+0000 7ff0ca992640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff0c4118e90 0x7ff0c41192a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.810+0000 7ff0ca992640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0c4114b90 con 0x7ff0c4114070 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.810+0000 7ff0ca992640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff0c4114d20 con 0x7ff0c4118e90 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff0c4118e90 0x7ff0c41192a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff0c4118e90 0x7ff0c41192a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45836/0 (socket says 172.21.15.152:45836) 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 -- 172.21.15.152:0/1284119806 learned_addr learned my addr 172.21.15.152:0/1284119806 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 -- 172.21.15.152:0/1284119806 >> v1:172.21.15.116:6789/0 conn(0x7ff0c4114070 legacy=0x7ff0c4114480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 -- 172.21.15.152:0/1284119806 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c4114e90 con 0x7ff0c4118e90 2024-08-16T02:36:58.812 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c3fff640 1 --2- 172.21.15.152:0/1284119806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4118e90 0x7ff0c41192a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f60e56ba770ea3 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c2ffd640 1 -- 172.21.15.152:0/1284119806 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0b4009e50 con 0x7ff0c4118e90 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c2ffd640 1 -- 172.21.15.152:0/1284119806 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff0b400e230 con 0x7ff0c4118e90 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.811+0000 7ff0c2ffd640 1 -- 172.21.15.152:0/1284119806 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0b400e4d0 con 0x7ff0c4118e90 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 -- 172.21.15.152:0/1284119806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4118e90 msgr2=0x7ff0c41192a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 --2- 172.21.15.152:0/1284119806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4118e90 0x7ff0c41192a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 -- 172.21.15.152:0/1284119806 shutdown_connections 2024-08-16T02:36:58.814 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 --2- 172.21.15.152:0/1284119806 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4118e90 0x7ff0c41192a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 -- 172.21.15.152:0/1284119806 >> 172.21.15.152:0/1284119806 conn(0x7ff0c406f810 msgr2=0x7ff0c406fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 -- 172.21.15.152:0/1284119806 shutdown_connections 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.812+0000 7ff0ca992640 1 -- 172.21.15.152:0/1284119806 wait complete. 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.813+0000 7ff0ca992640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.813+0000 7ff0ca992640 1 Processor -- start 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.813+0000 7ff0ca992640 1 -- start start 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0ca992640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:58.815 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0ca992640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0c41a93c0 con 0x7ff0c4114070 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0c3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45852/0 (socket says 172.21.15.152:45852) 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0c3fff640 1 -- 172.21.15.152:0/512052268 learned_addr learned my addr 172.21.15.152:0/512052268 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.814+0000 7ff0c3fff640 1 -- 172.21.15.152:0/512052268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0c41a9500 con 0x7ff0c4114070 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c3fff640 1 --2- 172.21.15.152:0/512052268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0b400eea0 con 0x7ff0c4114070 2024-08-16T02:36:58.817 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff0b4015420 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0c41a9750 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff0b40156c0 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0c41ae650 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7ff0b4015c00 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.815+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 862+0+0 (crc 0 0 0) 0x7ff0b4011740 con 0x7ff0c4114070 2024-08-16T02:36:58.818 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.816+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0c4077700 con 0x7ff0c4114070 2024-08-16T02:36:58.819 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.818+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7ff0b40119a0 con 0x7ff0c4114070 2024-08-16T02:36:58.893 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.893+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d5599bff-f9e3-4704-a9c2-f90504977470", "id": 1} v 0) -- 0x7ff0c4120090 con 0x7ff0c4114070 2024-08-16T02:36:58.895 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.894+0000 7ff0c0ff9640 1 -- 172.21.15.152:0/512052268 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d5599bff-f9e3-4704-a9c2-f90504977470", "id": 1}]=0 v5) ==== 112+0+2 (crc 0 0 0) 0x7ff0b4011b40 con 0x7ff0c4114070 2024-08-16T02:36:58.895 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 msgr2=0x7ff0c41a8df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 --2- 172.21.15.152:0/512052268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 shutdown_connections 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 --2- 172.21.15.152:0/512052268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0c4114070 0x7ff0c41a8df0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 >> 172.21.15.152:0/512052268 conn(0x7ff0c406f810 msgr2=0x7ff0c4117a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:58.896 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 shutdown_connections 2024-08-16T02:36:58.897 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:58.895+0000 7ff0ca992640 1 -- 172.21.15.152:0/512052268 wait complete. 2024-08-16T02:36:58.907 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 08298ee2-d217-4571-847a-605bd52c18d7 2 2024-08-16T02:36:59.014 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.008+0000 7f0fffaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.014+0000 7f0fffaed640 1 Processor -- start 2024-08-16T02:36:59.015 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0fffaed640 1 -- start start 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0fffaed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ff811cdd0 0x7f0ff811f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0fffaed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ff811f7c0 con 0x7f0ff8118e90 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0fffaed640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0ff8120980 con 0x7f0ff811cdd0 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0ffd088640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ff811cdd0 0x7f0ff811f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0ffd088640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0ff811cdd0 0x7f0ff811f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45862/0 (socket says 172.21.15.152:45862) 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0ffd088640 1 -- 172.21.15.152:0/900199954 learned_addr learned my addr 172.21.15.152:0/900199954 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.015+0000 7f0ffd889640 1 --1- 172.21.15.152:0/900199954 >> v1:172.21.15.116:6789/0 conn(0x7f0ff8118e90 0x7f0ff81192a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.152:37414/0 (socket says 172.21.15.152:37414) 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffd088640 1 -- 172.21.15.152:0/900199954 >> v1:172.21.15.116:6789/0 conn(0x7f0ff8118e90 legacy=0x7f0ff81192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffd088640 1 -- 172.21.15.152:0/900199954 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff8120af0 con 0x7f0ff811cdd0 2024-08-16T02:36:59.017 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffd088640 1 --2- 172.21.15.152:0/900199954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff811cdd0 0x7f0ff811f1f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7ef40b51973f71 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffc887640 1 -- 172.21.15.152:0/900199954 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0ff4009e50 con 0x7f0ff811cdd0 2024-08-16T02:36:59.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffc887640 1 -- 172.21.15.152:0/900199954 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f0ff400e230 con 0x7f0ff811cdd0 2024-08-16T02:36:59.018 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.016+0000 7f0ffc887640 1 -- 172.21.15.152:0/900199954 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0ff400e4d0 con 0x7f0ff811cdd0 2024-08-16T02:36:59.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 -- 172.21.15.152:0/900199954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff811cdd0 msgr2=0x7f0ff811f1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.020 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 --2- 172.21.15.152:0/900199954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff811cdd0 0x7f0ff811f1f0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 -- 172.21.15.152:0/900199954 shutdown_connections 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 --2- 172.21.15.152:0/900199954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff811cdd0 0x7f0ff811f1f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 -- 172.21.15.152:0/900199954 >> 172.21.15.152:0/900199954 conn(0x7f0ff806f810 msgr2=0x7f0ff806fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 -- 172.21.15.152:0/900199954 shutdown_connections 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 1 -- 172.21.15.152:0/900199954 wait complete. 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.017+0000 7f0fffaed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0fffaed640 1 Processor -- start 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0fffaed640 1 -- start start 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0fffaed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0fffaed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff81a9330 con 0x7f0ff8118e90 2024-08-16T02:36:59.021 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0ffd889640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0ffd889640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45876/0 (socket says 172.21.15.152:45876) 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.018+0000 7f0ffd889640 1 -- 172.21.15.152:0/1559225322 learned_addr learned my addr 172.21.15.152:0/1559225322 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0ffd889640 1 -- 172.21.15.152:0/1559225322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff81a9470 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0ffd889640 1 --2- 172.21.15.152:0/1559225322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0fec0059a0 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f0fec005b00 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff81a9720 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f0fec005da0 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff81ae640 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f0fec00cba0 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.019+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 977+0+0 (crc 0 0 0) 0x7f0fec00fad0 con 0x7f0ff8118e90 2024-08-16T02:36:59.022 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.020+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff8077700 con 0x7f0ff8118e90 2024-08-16T02:36:59.023 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.023+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f0fec00f230 con 0x7f0ff8118e90 2024-08-16T02:36:59.098 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.098+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "08298ee2-d217-4571-847a-605bd52c18d7", "id": 2} v 0) -- 0x7f0ff811cdd0 con 0x7f0ff8118e90 2024-08-16T02:36:59.099 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.099+0000 7f0fea7fc640 1 -- 172.21.15.152:0/1559225322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08298ee2-d217-4571-847a-605bd52c18d7", "id": 2}]=0 v6) ==== 112+0+2 (crc 0 0 0) 0x7f0fec00fe00 con 0x7f0ff8118e90 2024-08-16T02:36:59.099 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-08-16T02:36:59.100 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 msgr2=0x7f0ff81a8d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 --2- 172.21.15.152:0/1559225322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 shutdown_connections 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 --2- 172.21.15.152:0/1559225322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ff8118e90 0x7f0ff81a8d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 >> 172.21.15.152:0/1559225322 conn(0x7f0ff806f810 msgr2=0x7f0ff8110d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 shutdown_connections 2024-08-16T02:36:59.101 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.100+0000 7f0fffaed640 1 -- 172.21.15.152:0/1559225322 wait complete. 2024-08-16T02:36:59.111 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new f095864e-6d26-4aeb-9a17-b49b16ab8210 3 2024-08-16T02:36:59.218 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.212+0000 7f98e4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.218+0000 7f98e4e6d640 1 Processor -- start 2024-08-16T02:36:59.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.218+0000 7f98e4e6d640 1 -- start start 2024-08-16T02:36:59.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98e4e6d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98e011cdd0 0x7f98e011f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.219 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98e4e6d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e011f7c0 con 0x7f98e0118e90 2024-08-16T02:36:59.220 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98e4e6d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f98e0120980 con 0x7f98e011cdd0 2024-08-16T02:36:59.221 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98ddd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98e011cdd0 0x7f98e011f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.221 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98ddd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f98e011cdd0 0x7f98e011f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45892/0 (socket says 172.21.15.152:45892) 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98ddd9b640 1 -- 172.21.15.152:0/3596323674 learned_addr learned my addr 172.21.15.152:0/3596323674 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98ddd9b640 1 -- 172.21.15.152:0/3596323674 >> v1:172.21.15.116:6789/0 conn(0x7f98e0118e90 legacy=0x7f98e01192a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.219+0000 7f98ddd9b640 1 -- 172.21.15.152:0/3596323674 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e0120af0 con 0x7f98e011cdd0 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98ddd9b640 1 --2- 172.21.15.152:0/3596323674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e011cdd0 0x7f98e011f1f0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62b5e6439d92c8b server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98dd59a640 1 -- 172.21.15.152:0/3596323674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f98d0009e50 con 0x7f98e011cdd0 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98dd59a640 1 -- 172.21.15.152:0/3596323674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f98d000e230 con 0x7f98e011cdd0 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98dd59a640 1 -- 172.21.15.152:0/3596323674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f98d000e4d0 con 0x7f98e011cdd0 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98e4e6d640 1 -- 172.21.15.152:0/3596323674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e011cdd0 msgr2=0x7f98e011f1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.220+0000 7f98e4e6d640 1 --2- 172.21.15.152:0/3596323674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e011cdd0 0x7f98e011f1f0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 1 -- 172.21.15.152:0/3596323674 shutdown_connections 2024-08-16T02:36:59.222 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 1 --2- 172.21.15.152:0/3596323674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e011cdd0 0x7f98e011f1f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 1 -- 172.21.15.152:0/3596323674 >> 172.21.15.152:0/3596323674 conn(0x7f98e006f810 msgr2=0x7f98e006fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 1 -- 172.21.15.152:0/3596323674 shutdown_connections 2024-08-16T02:36:59.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 1 -- 172.21.15.152:0/3596323674 wait complete. 2024-08-16T02:36:59.223 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.221+0000 7f98e4e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98e4e6d640 1 Processor -- start 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98e4e6d640 1 -- start start 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98e4e6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98e4e6d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f98e01a9330 con 0x7f98e0118e90 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.152:45896/0 (socket says 172.21.15.152:45896) 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.222+0000 7f98de59c640 1 -- 172.21.15.152:0/260185660 learned_addr learned my addr 172.21.15.152:0/260185660 (peer_addr_for_me v2:172.21.15.152:0/0) 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98de59c640 1 -- 172.21.15.152:0/260185660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e01a9470 con 0x7f98e0118e90 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98de59c640 1 --2- 172.21.15.152:0/260185660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f98d400c950 con 0x7f98e0118e90 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f98d400cab0 con 0x7f98e0118e90 2024-08-16T02:36:59.225 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f98d400cd50 con 0x7f98e0118e90 2024-08-16T02:36:59.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e01a9720 con 0x7f98e0118e90 2024-08-16T02:36:59.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e01ae640 con 0x7f98e0118e90 2024-08-16T02:36:59.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f98d40116a0 con 0x7f98e0118e90 2024-08-16T02:36:59.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.223+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1092+0+0 (crc 0 0 0) 0x7f98d400f230 con 0x7f98e0118e90 2024-08-16T02:36:59.226 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.224+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e0077700 con 0x7f98e0118e90 2024-08-16T02:36:59.227 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.227+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f98d4011b30 con 0x7f98e0118e90 2024-08-16T02:36:59.301 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.301+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f095864e-6d26-4aeb-9a17-b49b16ab8210", "id": 3} v 0) -- 0x7f98e011cdd0 con 0x7f98e0118e90 2024-08-16T02:36:59.303 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.303+0000 7f98ceffd640 1 -- 172.21.15.152:0/260185660 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f095864e-6d26-4aeb-9a17-b49b16ab8210", "id": 3}]=0 v7) ==== 112+0+2 (crc 0 0 0) 0x7f98d4011da0 con 0x7f98e0118e90 2024-08-16T02:36:59.303 INFO:teuthology.orchestra.run.smithi152.stdout:3 2024-08-16T02:36:59.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 msgr2=0x7f98e01a8d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 --2- 172.21.15.152:0/260185660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.304 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 shutdown_connections 2024-08-16T02:36:59.305 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 --2- 172.21.15.152:0/260185660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f98e0118e90 0x7f98e01a8d60 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.305 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 >> 172.21.15.152:0/260185660 conn(0x7f98e006f810 msgr2=0x7f98e0110d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.305 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 shutdown_connections 2024-08-16T02:36:59.305 INFO:teuthology.orchestra.run.smithi152.stderr:2024-08-16T02:36:59.304+0000 7f98e4e6d640 1 -- 172.21.15.152:0/260185660 wait complete. 2024-08-16T02:36:59.315 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-08-16T02:36:59.349 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-16T02:36:59.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-08-16T02:36:59.392 INFO:tasks.ceph.osd.0:Started 2024-08-16T02:36:59.392 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-08-16T02:36:59.424 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-16T02:36:59.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-08-16T02:36:59.467 INFO:tasks.ceph.osd.1:Started 2024-08-16T02:36:59.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-08-16T02:36:59.498 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-16T02:36:59.498 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-08-16T02:36:59.541 INFO:tasks.ceph.osd.2:Started 2024-08-16T02:36:59.541 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-08-16T02:36:59.569 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-16T02:36:59.569 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-08-16T02:36:59.612 INFO:tasks.ceph.osd.3:Started 2024-08-16T02:36:59.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:36:59.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.723+0000 7fb67fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.728+0000 7fb67fa6d640 1 Processor -- start 2024-08-16T02:36:59.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.728+0000 7fb67fa6d640 1 -- start start 2024-08-16T02:36:59.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67fa6d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb67809fe30 0x7fb6780a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67fa6d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6780a14c0 con 0x7fb6780a0780 2024-08-16T02:36:59.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67fa6d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb67806f380 con 0x7fb67809fe30 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d809640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb67809fe30 0x7fb6780a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d008640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb6780a0780 0x7fb678135390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44386/0 (socket says 172.21.15.116:44386) 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d809640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb67809fe30 0x7fb6780a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60282/0 (socket says 172.21.15.116:60282) 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d008640 1 -- 172.21.15.116:0/2260430848 learned_addr learned my addr 172.21.15.116:0/2260430848 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d809640 1 -- 172.21.15.116:0/2260430848 >> v1:172.21.15.116:6789/0 conn(0x7fb6780a0780 legacy=0x7fb678135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67c807640 1 -- 172.21.15.116:0/2260430848 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658535145 0 0) 0x7fb6780a14c0 con 0x7fb6780a0780 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.729+0000 7fb67d809640 1 -- 172.21.15.116:0/2260430848 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6780a2130 con 0x7fb67809fe30 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67d809640 1 --2- 172.21.15.116:0/2260430848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6780a0240 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff1e44c23bf9672 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67c807640 1 -- 172.21.15.116:0/2260430848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb674009d90 con 0x7fb67809fe30 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67c807640 1 -- 172.21.15.116:0/2260430848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb67400e230 con 0x7fb67809fe30 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67c807640 1 -- 172.21.15.116:0/2260430848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb67400e4d0 con 0x7fb67809fe30 2024-08-16T02:36:59.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67fa6d640 1 -- 172.21.15.116:0/2260430848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 msgr2=0x7fb6780a0240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67fa6d640 1 --2- 172.21.15.116:0/2260430848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6780a0240 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67fa6d640 1 -- 172.21.15.116:0/2260430848 shutdown_connections 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67fa6d640 1 --2- 172.21.15.116:0/2260430848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6780a0240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.730+0000 7fb67fa6d640 1 -- 172.21.15.116:0/2260430848 >> 172.21.15.116:0/2260430848 conn(0x7fb67806caf0 msgr2=0x7fb67806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.731+0000 7fb67fa6d640 1 -- 172.21.15.116:0/2260430848 shutdown_connections 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.731+0000 7fb67fa6d640 1 -- 172.21.15.116:0/2260430848 wait complete. 2024-08-16T02:36:59.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.731+0000 7fb67fa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.732+0000 7fb67fa6d640 1 Processor -- start 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.732+0000 7fb67fa6d640 1 -- start start 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.733+0000 7fb67fa6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.733+0000 7fb67fa6d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb678122230 con 0x7fb67809fe30 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.733+0000 7fb67d809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.733+0000 7fb67d809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60290/0 (socket says 172.21.15.116:60290) 2024-08-16T02:36:59.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.733+0000 7fb67d809640 1 -- 172.21.15.116:0/1436667102 learned_addr learned my addr 172.21.15.116:0/1436667102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb67d809640 1 -- 172.21.15.116:0/1436667102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb678124110 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb67d809640 1 --2- 172.21.15.116:0/1436667102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb674014620 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb674014780 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb678121e90 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb674014a20 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb678125870 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7fb674013400 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.734+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7fb674019040 con 0x7fb67809fe30 2024-08-16T02:36:59.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.735+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb678123f60 con 0x7fb67809fe30 2024-08-16T02:36:59.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.737+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7fb6740116a0 con 0x7fb67809fe30 2024-08-16T02:36:59.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.800+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb6780a0240 con 0x7fb67809fe30 2024-08-16T02:36:59.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.801+0000 7fb66e7fc640 1 -- 172.21.15.116:0/1436667102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7fb6740118f0 con 0x7fb67809fe30 2024-08-16T02:36:59.802 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:36:59.802 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":7,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:36:59.302189+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 msgr2=0x7fb6781356d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 --2- 172.21.15.116:0/1436667102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 shutdown_connections 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 --2- 172.21.15.116:0/1436667102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb67809fe30 0x7fb6781356d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 >> 172.21.15.116:0/1436667102 conn(0x7fb67806caf0 msgr2=0x7fb6781bb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 shutdown_connections 2024-08-16T02:36:59.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:36:59.802+0000 7fb67fa6d640 1 -- 172.21.15.116:0/1436667102 wait complete. 2024-08-16T02:36:59.813 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-16T02:36:59.813 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-16T02:37:00.216 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:37:00.321 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.316+0000 7f5b35192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:00.322 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.321+0000 7f5b35192640 1 Processor -- start 2024-08-16T02:37:00.322 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.321+0000 7f5b35192640 1 -- start start 2024-08-16T02:37:00.324 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.322+0000 7f5b35192640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b300a2110 0x7f5b3013a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:00.324 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b35192640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b3006ed70 con 0x7f5b300a15f0 2024-08-16T02:37:00.324 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b35192640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5b3006f820 con 0x7f5b300a2110 2024-08-16T02:37:00.324 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b300a2110 0x7f5b3013a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5b300a2110 0x7f5b3013a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60292/0 (socket says 172.21.15.116:60292) 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 -- 172.21.15.116:0/394668900 learned_addr learned my addr 172.21.15.116:0/394668900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5b300a15f0 0x7f5b300a1a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44400/0 (socket says 172.21.15.116:44400) 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 -- 172.21.15.116:0/394668900 >> v1:172.21.15.116:6789/0 conn(0x7f5b300a15f0 legacy=0x7f5b300a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2dd9b640 1 -- 172.21.15.116:0/394668900 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517102702 0 0) 0x7f5b3006ed70 con 0x7f5b300a15f0 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 -- 172.21.15.116:0/394668900 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b300a4340 con 0x7f5b300a2110 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2e59c640 1 --2- 172.21.15.116:0/394668900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a2110 0x7f5b3013a670 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef866d9786f3fc1f server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2dd9b640 1 -- 172.21.15.116:0/394668900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5b18009e50 con 0x7f5b300a2110 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2dd9b640 1 -- 172.21.15.116:0/394668900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5b1800e230 con 0x7f5b300a2110 2024-08-16T02:37:00.325 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.323+0000 7f5b2dd9b640 1 -- 172.21.15.116:0/394668900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5b1800e4d0 con 0x7f5b300a2110 2024-08-16T02:37:00.326 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 -- 172.21.15.116:0/394668900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a2110 msgr2=0x7f5b3013a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:00.326 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 --2- 172.21.15.116:0/394668900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a2110 0x7f5b3013a670 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:00.326 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 -- 172.21.15.116:0/394668900 shutdown_connections 2024-08-16T02:37:00.326 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 --2- 172.21.15.116:0/394668900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a2110 0x7f5b3013a670 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 -- 172.21.15.116:0/394668900 >> 172.21.15.116:0/394668900 conn(0x7f5b300a4e90 msgr2=0x7f5b3009fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 -- 172.21.15.116:0/394668900 shutdown_connections 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.324+0000 7f5b35192640 1 -- 172.21.15.116:0/394668900 wait complete. 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.325+0000 7f5b35192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.326+0000 7f5b35192640 1 Processor -- start 2024-08-16T02:37:00.327 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.326+0000 7f5b35192640 1 -- start start 2024-08-16T02:37:00.329 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.327+0000 7f5b35192640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:00.329 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.327+0000 7f5b35192640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5b301231b0 con 0x7f5b300a15f0 2024-08-16T02:37:00.329 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.327+0000 7f5b2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:00.329 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.327+0000 7f5b2ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60298/0 (socket says 172.21.15.116:60298) 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b2ed9d640 1 -- 172.21.15.116:0/1120965344 learned_addr learned my addr 172.21.15.116:0/1120965344 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b2ed9d640 1 -- 172.21.15.116:0/1120965344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b30126130 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b2ed9d640 1 --2- 172.21.15.116:0/1120965344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5b20004b80 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f5b20004ce0 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b30122bb0 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f5b20007120 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b30124ee0 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (crc 0 0 0) 0x7f5b2000c520 con 0x7f5b300a15f0 2024-08-16T02:37:00.330 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.328+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1207+0+0 (crc 0 0 0) 0x7f5b2000fc00 con 0x7f5b300a15f0 2024-08-16T02:37:00.331 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.329+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b300a1a00 con 0x7f5b300a15f0 2024-08-16T02:37:00.332 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.332+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (crc 0 0 0) 0x7f5b2000fe60 con 0x7f5b300a15f0 2024-08-16T02:37:00.410 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.409+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b3006bf10 con 0x7f5b300a15f0 2024-08-16T02:37:00.410 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.410+0000 7f5b277fe640 1 -- 172.21.15.116:0/1120965344 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v7) ==== 74+0+4152 (crc 0 0 0) 0x7f5b2000f230 con 0x7f5b300a15f0 2024-08-16T02:37:00.411 INFO:teuthology.misc.health.smithi116.stdout: 2024-08-16T02:37:00.411 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":7,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:36:59.302189+0000","last_up_change":"0.000000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 msgr2=0x7f5b30123600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 --2- 172.21.15.116:0/1120965344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 shutdown_connections 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 --2- 172.21.15.116:0/1120965344 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5b300a15f0 0x7f5b30123600 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 >> 172.21.15.116:0/1120965344 conn(0x7f5b300a4e90 msgr2=0x7f5b300716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 shutdown_connections 2024-08-16T02:37:00.412 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:00.411+0000 7f5b35192640 1 -- 172.21.15.116:0/1120965344 wait complete. 2024-08-16T02:37:00.421 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-08-16T02:37:02.670 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-08-16T02:37:02.669+0000 7f23afa75640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-08-16T02:37:06.825 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-16T02:37:06.927 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.922+0000 7f54e0352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:06.928 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.927+0000 7f54e0352640 1 Processor -- start 2024-08-16T02:37:06.928 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.927+0000 7f54e0352640 1 -- start start 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.928+0000 7f54e0352640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54d81332f0 0x7f54d8133700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.928+0000 7f54e0352640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d8072840 con 0x7f54d80a11f0 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.928+0000 7f54e0352640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f54d809ba20 con 0x7f54d81332f0 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54d81332f0 0x7f54d8133700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f54d81332f0 0x7f54d8133700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37656/0 (socket says 172.21.15.116:37656) 2024-08-16T02:37:06.930 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 -- 172.21.15.116:0/3690847277 learned_addr learned my addr 172.21.15.116:0/3690847277 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 -- 172.21.15.116:0/3690847277 >> v1:172.21.15.116:6789/0 conn(0x7f54d80a11f0 legacy=0x7f54d80a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 -- 172.21.15.116:0/3690847277 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d8074b50 con 0x7f54d81332f0 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54de0ee640 1 --2- 172.21.15.116:0/3690847277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d81332f0 0x7f54d8133700 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f1d8fd71f3cc6e server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54dd0ec640 1 -- 172.21.15.116:0/3690847277 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f54d4009e50 con 0x7f54d81332f0 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54dd0ec640 1 -- 172.21.15.116:0/3690847277 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f54d400e230 con 0x7f54d81332f0 2024-08-16T02:37:06.931 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.929+0000 7f54dd0ec640 1 -- 172.21.15.116:0/3690847277 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f54d400e4d0 con 0x7f54d81332f0 2024-08-16T02:37:06.932 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 -- 172.21.15.116:0/3690847277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d81332f0 msgr2=0x7f54d8133700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:06.932 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 --2- 172.21.15.116:0/3690847277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d81332f0 0x7f54d8133700 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 -- 172.21.15.116:0/3690847277 shutdown_connections 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 --2- 172.21.15.116:0/3690847277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d81332f0 0x7f54d8133700 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 -- 172.21.15.116:0/3690847277 >> 172.21.15.116:0/3690847277 conn(0x7f54d80a22e0 msgr2=0x7f54d80a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 -- 172.21.15.116:0/3690847277 shutdown_connections 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 1 -- 172.21.15.116:0/3690847277 wait complete. 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.930+0000 7f54e0352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.931+0000 7f54e0352640 1 Processor -- start 2024-08-16T02:37:06.933 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.932+0000 7f54e0352640 1 -- start start 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.932+0000 7f54e0352640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54e0352640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54d811ffe0 con 0x7f54d80a11f0 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54de0ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54de0ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37668/0 (socket says 172.21.15.116:37668) 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54de0ee640 1 -- 172.21.15.116:0/1285456892 learned_addr learned my addr 172.21.15.116:0/1285456892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:06.935 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54de0ee640 1 -- 172.21.15.116:0/1285456892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d809f180 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54de0ee640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f54d400ed20 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f54d400ee80 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f54d4015550 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d81228f0 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.933+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d8120690 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.934+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d8074380 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.934+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7f54d40156b0 con 0x7f54d80a11f0 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.934+0000 7f54ceffd640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 0x7f54bc0401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:06.936 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.934+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2319+0+0 (crc 0 0 0) 0x7f54d404faf0 con 0x7f54d80a11f0 2024-08-16T02:37:06.937 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.935+0000 7f54dd8ed640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 0x7f54bc0401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:06.937 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.935+0000 7f54dd8ed640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 0x7f54bc0401c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:06.944 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:06.944+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f54d4010a00 con 0x7f54d80a11f0 2024-08-16T02:37:07.136 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.135+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54d8076690 con 0x7f54d80a11f0 2024-08-16T02:37:07.136 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.136+0000 7f54ceffd640 1 -- 172.21.15.116:0/1285456892 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6979 (crc 0 0 0) 0x7f54d4010310 con 0x7f54d80a11f0 2024-08-16T02:37:07.137 INFO:teuthology.misc.health.smithi116.stdout: 2024-08-16T02:37:07.137 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":11,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:37:06.724297+0000","last_up_change":"2024-08-16T02:37:03.718029+0000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:37:05.689624+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":295590560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":295590560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":295590560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6805","nonce":295590560}]},"public_addr":"172.21.15.116:6804/295590560","cluster_addr":"172.21.15.116:6803/295590560","heartbeat_back_addr":"172.21.15.116:6806/295590560","heartbeat_front_addr":"172.21.15.116:6805/295590560","state":["exists","up"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2086482905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6801","nonce":2086482905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":2086482905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2086482905}]},"public_addr":"172.21.15.116:6802/2086482905","cluster_addr":"172.21.15.116:6801/2086482905","heartbeat_back_addr":"172.21.15.116:6807/2086482905","heartbeat_front_addr":"172.21.15.116:6808/2086482905","state":["exists","up"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1500775361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":1500775361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1500775361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":1500775361}]},"public_addr":"172.21.15.152:6820/1500775361","cluster_addr":"172.21.15.152:6821/1500775361","heartbeat_back_addr":"172.21.15.152:6806/1500775361","heartbeat_front_addr":"172.21.15.152:6807/1500775361","state":["exists","up"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2721550083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2721550083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2721550083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2721550083}]},"public_addr":"172.21.15.152:6804/2721550083","cluster_addr":"172.21.15.152:6803/2721550083","heartbeat_back_addr":"172.21.15.152:6805/2721550083","heartbeat_front_addr":"172.21.15.152:6802/2721550083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:37:07.139 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 msgr2=0x7f54bc0401c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 0x7f54bc0401c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 msgr2=0x7f54d811eca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 shutdown_connections 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f54bc03dcb0 0x7f54bc0401c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 --2- 172.21.15.116:0/1285456892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54d80a11f0 0x7f54d811eca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 >> 172.21.15.116:0/1285456892 conn(0x7f54d80a22e0 msgr2=0x7f54d80a35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 shutdown_connections 2024-08-16T02:37:07.140 INFO:teuthology.misc.health.smithi116.stderr:2024-08-16T02:37:07.138+0000 7f54e0352640 1 -- 172.21.15.116:0/1285456892 wait complete. 2024-08-16T02:37:07.149 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-08-16T02:37:07.149 INFO:tasks.ceph:Creating RBD pool 2024-08-16T02:37:07.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-16T02:37:07.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.251+0000 7f6e33a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:07.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.259+0000 7f6e33a2d640 1 Processor -- start 2024-08-16T02:37:07.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.259+0000 7f6e33a2d640 1 -- start start 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e33a2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e2c0a0540 0x7f6e2c13a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e33a2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2c0a7570 con 0x7f6e2c0a6ea0 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e33a2d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c06f380 con 0x7f6e2c0a0540 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e30fc8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e2c0a0540 0x7f6e2c13a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e30fc8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e2c0a0540 0x7f6e2c13a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37682/0 (socket says 172.21.15.116:37682) 2024-08-16T02:37:07.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e30fc8640 1 -- 172.21.15.116:0/4261961525 learned_addr learned my addr 172.21.15.116:0/4261961525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:07.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.260+0000 7f6e317c9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6e2c0a6ea0 0x7f6e2c09fe30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49050/0 (socket says 172.21.15.116:49050) 2024-08-16T02:37:07.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.261+0000 7f6e30fc8640 1 -- 172.21.15.116:0/4261961525 >> v1:172.21.15.116:6789/0 conn(0x7f6e2c0a6ea0 legacy=0x7f6e2c09fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.261+0000 7f6e30fc8640 1 -- 172.21.15.116:0/4261961525 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c0a1630 con 0x7f6e2c0a0540 2024-08-16T02:37:07.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.261+0000 7f6e30fc8640 1 --2- 172.21.15.116:0/4261961525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c13a770 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f3e204e72849f3 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:07.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.326+0000 7f6e23fff640 1 -- 172.21.15.116:0/4261961525 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6e1c009e50 con 0x7f6e2c0a0540 2024-08-16T02:37:07.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.326+0000 7f6e23fff640 1 -- 172.21.15.116:0/4261961525 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6e1c00e230 con 0x7f6e2c0a0540 2024-08-16T02:37:07.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.326+0000 7f6e23fff640 1 -- 172.21.15.116:0/4261961525 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6e1c00e4d0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 -- 172.21.15.116:0/4261961525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 msgr2=0x7f6e2c13a770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/4261961525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c13a770 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 -- 172.21.15.116:0/4261961525 shutdown_connections 2024-08-16T02:37:07.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/4261961525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c13a770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 -- 172.21.15.116:0/4261961525 >> 172.21.15.116:0/4261961525 conn(0x7f6e2c06ce60 msgr2=0x7f6e2c06d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:07.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 -- 172.21.15.116:0/4261961525 shutdown_connections 2024-08-16T02:37:07.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 1 -- 172.21.15.116:0/4261961525 wait complete. 2024-08-16T02:37:07.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.327+0000 7f6e33a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:07.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.328+0000 7f6e33a2d640 1 Processor -- start 2024-08-16T02:37:07.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.328+0000 7f6e33a2d640 1 -- start start 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.329+0000 7f6e33a2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.329+0000 7f6e33a2d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c1249b0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e317c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e317c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37694/0 (socket says 172.21.15.116:37694) 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e317c9640 1 -- 172.21.15.116:0/409686205 learned_addr learned my addr 172.21.15.116:0/409686205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:07.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e317c9640 1 -- 172.21.15.116:0/409686205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c124170 con 0x7f6e2c0a0540 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e317c9640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6e280059a0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f6e28005b00 con 0x7f6e2c0a0540 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2c125260 con 0x7f6e2c0a0540 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.330+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6e28005da0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.331+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2c1259e0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.331+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7f6e280070b0 con 0x7f6e2c0a0540 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.332+0000 7f6e21ffb640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 0x7f6e10040190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.332+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(11..11 src has 1..11) ==== 2319+0+0 (crc 0 0 0) 0x7f6e2804e460 con 0x7f6e2c0a0540 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.332+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e2c127010 con 0x7f6e2c0a0540 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.332+0000 7f6e30fc8640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 0x7f6e10040190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.332+0000 7f6e30fc8640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 0x7f6e10040190 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:07.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.337+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f6e2c127010 con 0x7f6e2c0a0540 2024-08-16T02:37:07.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.529+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f6e2c127010 con 0x7f6e2c0a0540 2024-08-16T02:37:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.728+0000 7f6e21ffb640 1 -- 172.21.15.116:0/409686205 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v12) ==== 109+0+0 (crc 0 0 0) 0x7f6e2c127010 con 0x7f6e2c0a0540 2024-08-16T02:37:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'rbd' created 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 msgr2=0x7f6e10040190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 0x7f6e10040190 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 msgr2=0x7f6e2c122f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 shutdown_connections 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f6e1003dc80 0x7f6e10040190 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 --2- 172.21.15.116:0/409686205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e2c0a0540 0x7f6e2c122f30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 >> 172.21.15.116:0/409686205 conn(0x7f6e2c06ce60 msgr2=0x7f6e2c0a30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 shutdown_connections 2024-08-16T02:37:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.731+0000 7f6e33a2d640 1 -- 172.21.15.116:0/409686205 wait complete. 2024-08-16T02:37:07.745 DEBUG:teuthology.orchestra.run.smithi116:> rbd --cluster ceph pool init rbd 2024-08-16T02:37:07.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.825+0000 7ffb5e8458c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.825+0000 7ffb5e8458c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.827+0000 7ffb5e8458c0 1 Processor -- start 2024-08-16T02:37:07.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.827+0000 7ffb5e8458c0 1 -- start start 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb5e8458c0 1 --2- >> v2:172.21.15.116:3300/0 conn(0x29fd700 0x2a90f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb5e8458c0 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x285e500 con 0x2a91490 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb5e8458c0 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x289d130 con 0x29fd700 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb58a20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x29fd700 0x2a90f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb58a20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x29fd700 0x2a90f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37710/0 (socket says 172.21.15.116:37710) 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb58a20640 1 -- 172.21.15.116:0/1791044113 learned_addr learned my addr 172.21.15.116:0/1791044113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb53fff640 1 --1- 172.21.15.116:0/1791044113 >> v1:172.21.15.116:6789/0 conn(0x2a91490 0x2a8dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49060/0 (socket says 172.21.15.116:49060) 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb58a20640 1 -- 172.21.15.116:0/1791044113 >> v1:172.21.15.116:6789/0 conn(0x2a91490 legacy=0x2a8dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.828+0000 7ffb58a20640 1 -- 172.21.15.116:0/1791044113 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x28343d0 con 0x29fd700 2024-08-16T02:37:07.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.829+0000 7ffb58a20640 1 --2- 172.21.15.116:0/1791044113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a90f50 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82aad2c3af3d4896 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:07.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.830+0000 7ffb537fe640 1 -- 172.21.15.116:0/1791044113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb4802c5e0 con 0x29fd700 2024-08-16T02:37:07.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.830+0000 7ffb537fe640 1 -- 172.21.15.116:0/1791044113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ffb4802c740 con 0x29fd700 2024-08-16T02:37:07.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.830+0000 7ffb537fe640 1 -- 172.21.15.116:0/1791044113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb4802c9e0 con 0x29fd700 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1791044113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 msgr2=0x2a90f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1791044113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a90f50 crc :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1791044113 shutdown_connections 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1791044113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a90f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1791044113 >> 172.21.15.116:0/1791044113 conn(0x2847490 msgr2=0x29e36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1791044113 shutdown_connections 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1791044113 wait complete. 2024-08-16T02:37:07.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.831+0000 7ffb5e8458c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:07.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.832+0000 7ffb5e8458c0 1 Processor -- start 2024-08-16T02:37:07.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.832+0000 7ffb5e8458c0 1 -- start start 2024-08-16T02:37:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.833+0000 7ffb5e8458c0 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.833+0000 7ffb5e8458c0 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x28a7c10 con 0x29fd700 2024-08-16T02:37:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.833+0000 7ffb58a20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.833+0000 7ffb58a20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37722/0 (socket says 172.21.15.116:37722) 2024-08-16T02:37:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.833+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 learned_addr learned my addr 172.21.15.116:0/1197591195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x2842c50 con 0x29fd700 2024-08-16T02:37:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb58a20640 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb48006590 con 0x29fd700 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ffb480066f0 con 0x29fd700 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ffb48006990 con 0x29fd700 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x2834760 con 0x29fd700 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.834+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x28341c0 con 0x29fd700 2024-08-16T02:37:07.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.835+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 49824+0+0 (crc 0 0 0) 0x7ffb48006b40 con 0x29fd700 2024-08-16T02:37:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.835+0000 7ffb50ff9640 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 0x7ffb3803ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.835+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2661+0+0 (crc 0 0 0) 0x7ffb48031040 con 0x29fd700 2024-08-16T02:37:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.835+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x29fe550 con 0x29fd700 2024-08-16T02:37:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.836+0000 7ffb53fff640 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 0x7ffb3803ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:07.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:07.836+0000 7ffb53fff640 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 0x7ffb3803ea20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:08.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.609+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7ffb48034d30 con 0x29fd700 2024-08-16T02:37:08.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.739+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v13) ==== 141+0+0 (crc 0 0 0) 0x7ffb48006e40 con 0x29fd700 2024-08-16T02:37:08.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.739+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x285f6a0 con 0x29fd700 2024-08-16T02:37:08.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.745+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 8 ==== mon_get_version_reply(handle=1 version=13) ==== 24+0+0 (crc 0 0 0) 0x7ffb48033eb0 con 0x29fd700 2024-08-16T02:37:08.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.745+0000 7ffb52ffd640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=13}) -- 0x7ffb30002ae0 con 0x29fd700 2024-08-16T02:37:08.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.745+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 9 ==== osd_map(13..13 src has 1..13) ==== 658+0+0 (crc 0 0 0) 0x7ffb4802d650 con 0x29fd700 2024-08-16T02:37:08.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.745+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 0x2aa3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:08.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.745+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> v2:172.21.15.116:6802/2086482905 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13) -- 0x2a7ec20 con 0x2a91490 2024-08-16T02:37:08.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.749+0000 7ffb59c82640 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 0x2aa3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:08.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.767+0000 7ffb59c82640 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 0x2aa3380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:08.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.856+0000 7ffb59c82640 1 -- 172.21.15.116:0/1197591195 <== osd.1 v2:172.21.15.116:6802/2086482905 1 ==== osd_op_reply(1 rbd_trash [create] v13'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7ffb4c00a4b0 con 0x2a91490 2024-08-16T02:37:08.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.859+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 0x2af83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:08.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.859+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> v2:172.21.15.116:6804/295590560 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x2af89a0 con 0x2af5ef0 2024-08-16T02:37:08.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.860+0000 7ffb58a20640 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 0x2af83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:08.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.863+0000 7ffb58a20640 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 0x2af83d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:08.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.868+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 <== osd.0 v2:172.21.15.116:6804/295590560 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ffb48072040 con 0x2af5ef0 2024-08-16T02:37:08.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.868+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 --> v2:172.21.15.116:6804/295590560 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e13) -- 0x2af53b0 con 0x2af5ef0 2024-08-16T02:37:08.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.871+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 <== osd.0 v2:172.21.15.116:6804/295590560 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7ffb48072040 con 0x2af5ef0 2024-08-16T02:37:08.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:08.871+0000 7ffb527fc640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7ffb34001f60 con 0x29fd700 2024-08-16T02:37:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.752+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 10 ==== pool_op_reply(tid 4 (0) Success v14) ==== 55+0+0 (crc 0 0 0) 0x7ffb4802d1f0 con 0x29fd700 2024-08-16T02:37:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.752+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=14}) -- 0x7ffb38041940 con 0x29fd700 2024-08-16T02:37:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.753+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 11 ==== osd_map(14..14 src has 1..14) ==== 626+0+0 (crc 0 0 0) 0x7ffb48006df0 con 0x29fd700 2024-08-16T02:37:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.753+0000 7ffb52ffd640 1 -- 172.21.15.116:0/1197591195 --> v2:172.21.15.116:6804/295590560 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14) -- 0x7ffb30005640 con 0x2af5ef0 2024-08-16T02:37:09.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.790+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 <== osd.0 v2:172.21.15.116:6804/295590560 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v14'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7ffb48072040 con 0x2af5ef0 2024-08-16T02:37:09.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:09.790+0000 7ffb527fc640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v14) -- 0x7ffb34003370 con 0x29fd700 2024-08-16T02:37:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.754+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 12 ==== pool_op_reply(tid 6 (0) Success v15) ==== 43+0+0 (crc 0 0 0) 0x7ffb4802d420 con 0x29fd700 2024-08-16T02:37:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.754+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=15}) -- 0x7ffb38041b80 con 0x29fd700 2024-08-16T02:37:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.755+0000 7ffb50ff9640 1 -- 172.21.15.116:0/1197591195 <== mon.0 v2:172.21.15.116:3300/0 13 ==== osd_map(15..15 src has 1..15) ==== 654+0+0 (crc 0 0 0) 0x7ffb48037c80 con 0x29fd700 2024-08-16T02:37:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.755+0000 7ffb52ffd640 1 -- 172.21.15.116:0/1197591195 --> v2:172.21.15.116:6804/295590560 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15) -- 0x7ffb300059e0 con 0x2af5ef0 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb58a20640 1 -- 172.21.15.116:0/1197591195 <== osd.0 v2:172.21.15.116:6804/295590560 4 ==== osd_op_reply(7 rbd_info [write 0~19] v15'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7ffb48072040 con 0x2af5ef0 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 msgr2=0x2af83d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 0x2af83d0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 msgr2=0x2aa3380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 0x2aa3380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 msgr2=0x7ffb3803ea20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 0x7ffb3803ea20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 msgr2=0x2a7b900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 crc :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 shutdown_connections 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6804/295590560 conn(0x2af5ef0 0x2af83d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.793+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> v2:172.21.15.116:6802/2086482905 conn(0x2a91490 0x2aa3380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.794+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ffb3803c510 0x7ffb3803ea20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.794+0000 7ffb5e8458c0 1 --2- 172.21.15.116:0/1197591195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x29fd700 0x2a7b900 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.794+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 >> 172.21.15.116:0/1197591195 conn(0x2847490 msgr2=0x2a909d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.794+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 shutdown_connections 2024-08-16T02:37:10.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.794+0000 7ffb5e8458c0 1 -- 172.21.15.116:0/1197591195 wait complete. 2024-08-16T02:37:10.799 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-16T02:37:10.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-08-16T02:37:10.800 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-16T02:37:10.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.904+0000 7f928e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.910+0000 7f928e6ad640 1 Processor -- start 2024-08-16T02:37:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.910+0000 7f928e6ad640 1 -- start start 2024-08-16T02:37:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f928e6ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f928809d650 0x7f9288137530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:10.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f928e6ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92880a0260 con 0x7f928809cb30 2024-08-16T02:37:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f928e6ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9288070900 con 0x7f928809d650 2024-08-16T02:37:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f92877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f928809d650 0x7f9288137530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f9287fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f928809cb30 0x7f928809cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44884/0 (socket says 172.21.15.116:44884) 2024-08-16T02:37:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f92877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f928809d650 0x7f9288137530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47400/0 (socket says 172.21.15.116:47400) 2024-08-16T02:37:10.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f9287fff640 1 -- 172.21.15.116:0/1892706128 learned_addr learned my addr 172.21.15.116:0/1892706128 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f92877fe640 1 -- 172.21.15.116:0/1892706128 >> v1:172.21.15.116:6789/0 conn(0x7f928809cb30 legacy=0x7f928809cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f92877fe640 1 -- 172.21.15.116:0/1892706128 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f928806f710 con 0x7f928809d650 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f92877fe640 1 --2- 172.21.15.116:0/1892706128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809d650 0x7f9288137530 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb17a8ed304f865 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f9286ffd640 1 -- 172.21.15.116:0/1892706128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f927c009d90 con 0x7f928809d650 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.911+0000 7f9286ffd640 1 -- 172.21.15.116:0/1892706128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f927c00e230 con 0x7f928809d650 2024-08-16T02:37:10.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f9286ffd640 1 -- 172.21.15.116:0/1892706128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f927c00e4d0 con 0x7f928809d650 2024-08-16T02:37:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 -- 172.21.15.116:0/1892706128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809d650 msgr2=0x7f9288137530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 --2- 172.21.15.116:0/1892706128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809d650 0x7f9288137530 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 -- 172.21.15.116:0/1892706128 shutdown_connections 2024-08-16T02:37:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 --2- 172.21.15.116:0/1892706128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809d650 0x7f9288137530 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:10.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 -- 172.21.15.116:0/1892706128 >> 172.21.15.116:0/1892706128 conn(0x7f92880a09c0 msgr2=0x7f92880a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 -- 172.21.15.116:0/1892706128 shutdown_connections 2024-08-16T02:37:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.912+0000 7f928e6ad640 1 -- 172.21.15.116:0/1892706128 wait complete. 2024-08-16T02:37:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.913+0000 7f928e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.914+0000 7f928e6ad640 1 Processor -- start 2024-08-16T02:37:10.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.914+0000 7f928e6ad640 1 -- start start 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f928e6ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f928e6ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f92881204d0 con 0x7f928809cb30 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f9287fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f9287fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47410/0 (socket says 172.21.15.116:47410) 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f9287fff640 1 -- 172.21.15.116:0/3412141545 learned_addr learned my addr 172.21.15.116:0/3412141545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:10.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.915+0000 7f9287fff640 1 -- 172.21.15.116:0/3412141545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92881219a0 con 0x7f928809cb30 2024-08-16T02:37:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f9287fff640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9278005a60 con 0x7f928809cb30 2024-08-16T02:37:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f9278005bc0 con 0x7f928809cb30 2024-08-16T02:37:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92881201c0 con 0x7f928809cb30 2024-08-16T02:37:10.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f9278005e60 con 0x7f928809cb30 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.916+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9288123280 con 0x7f928809cb30 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.917+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f928809cf40 con 0x7f928809cb30 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.917+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f92780070b0 con 0x7f928809cb30 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.917+0000 7f9284ff9640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 0x7f9274040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.917+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f927804e740 con 0x7f928809cb30 2024-08-16T02:37:10.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.917+0000 7f92877fe640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 0x7f9274040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.918+0000 7f92877fe640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 0x7f9274040220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:10.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:10.922+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f927800d7f0 con 0x7f928809cb30 2024-08-16T02:37:11.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.108+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7f9288076690 con 0x7f928809cb30 2024-08-16T02:37:11.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.108+0000 7f9284ff9640 1 -- 172.21.15.116:0/3412141545 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v1) ==== 86+0+61 (crc 0 0 0) 0x7f927800d610 con 0x7f928809cb30 2024-08-16T02:37:11.109 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.110+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 msgr2=0x7f9274040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.110+0000 7f928e6ad640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 0x7f9274040220 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.110+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 msgr2=0x7f9288120900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.110+0000 7f928e6ad640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 shutdown_connections 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f927403dd10 0x7f9274040220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 --2- 172.21.15.116:0/3412141545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f928809cb30 0x7f9288120900 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 >> 172.21.15.116:0/3412141545 conn(0x7f92880a09c0 msgr2=0x7f92881ba230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 shutdown_connections 2024-08-16T02:37:11.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.111+0000 7f928e6ad640 1 -- 172.21.15.116:0/3412141545 wait complete. 2024-08-16T02:37:11.122 INFO:teuthology.orchestra.run.smithi116.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-08-16T02:37:11.122 INFO:tasks.ceph_manager:config epoch is 1 2024-08-16T02:37:11.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-16T02:37:11.123 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-16T02:37:11.123 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-16T02:37:11.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.225+0000 7fcb85b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.230+0000 7fcb85b52640 1 Processor -- start 2024-08-16T02:37:11.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.230+0000 7fcb85b52640 1 -- start start 2024-08-16T02:37:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.231+0000 7fcb85b52640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb8009cb30 0x7fcb8009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.231+0000 7fcb85b52640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb800a0260 con 0x7fcb8009d480 2024-08-16T02:37:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.231+0000 7fcb85b52640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcb80070900 con 0x7fcb8009cb30 2024-08-16T02:37:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb8009cb30 0x7fcb8009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcb8009cb30 0x7fcb8009cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47414/0 (socket says 172.21.15.116:47414) 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 -- 172.21.15.116:0/1606570457 learned_addr learned my addr 172.21.15.116:0/1606570457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcb8009d480 0x7fcb801322f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44894/0 (socket says 172.21.15.116:44894) 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 -- 172.21.15.116:0/1606570457 >> v1:172.21.15.116:6789/0 conn(0x7fcb8009d480 legacy=0x7fcb801322f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 -- 172.21.15.116:0/1606570457 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8006f710 con 0x7fcb8009cb30 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7f7fe640 1 --2- 172.21.15.116:0/1606570457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb8009cf40 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ebf103fc8c7629 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7e7fc640 1 -- 172.21.15.116:0/1606570457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcb70009e50 con 0x7fcb8009cb30 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7e7fc640 1 -- 172.21.15.116:0/1606570457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcb7000e230 con 0x7fcb8009cb30 2024-08-16T02:37:11.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.232+0000 7fcb7e7fc640 1 -- 172.21.15.116:0/1606570457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcb7000e4d0 con 0x7fcb8009cb30 2024-08-16T02:37:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 -- 172.21.15.116:0/1606570457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 msgr2=0x7fcb8009cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 --2- 172.21.15.116:0/1606570457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb8009cf40 crc :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 -- 172.21.15.116:0/1606570457 shutdown_connections 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 --2- 172.21.15.116:0/1606570457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb8009cf40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 -- 172.21.15.116:0/1606570457 >> 172.21.15.116:0/1606570457 conn(0x7fcb800a09c0 msgr2=0x7fcb800a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 -- 172.21.15.116:0/1606570457 shutdown_connections 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 1 -- 172.21.15.116:0/1606570457 wait complete. 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.233+0000 7fcb85b52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.234+0000 7fcb85b52640 1 Processor -- start 2024-08-16T02:37:11.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.235+0000 7fcb85b52640 1 -- start start 2024-08-16T02:37:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb85b52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb85b52640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb80133350 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47430/0 (socket says 172.21.15.116:47430) 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb7f7fe640 1 -- 172.21.15.116:0/551634527 learned_addr learned my addr 172.21.15.116:0/551634527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb7f7fe640 1 -- 172.21.15.116:0/551634527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb801203e0 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb7f7fe640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcb7000ee20 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fcb70015420 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb801217f0 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.236+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fcb700156c0 con 0x7fcb8009cb30 2024-08-16T02:37:11.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.237+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb80120040 con 0x7fcb8009cb30 2024-08-16T02:37:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.237+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb80122670 con 0x7fcb8009cb30 2024-08-16T02:37:11.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.238+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fcb70015870 con 0x7fcb8009cb30 2024-08-16T02:37:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.238+0000 7fcb67fff640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 0x7fcb6c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.238+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fcb7004fd50 con 0x7fcb8009cb30 2024-08-16T02:37:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.238+0000 7fcb7effd640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 0x7fcb6c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.239+0000 7fcb7effd640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 0x7fcb6c040220 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.243+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fcb70010a00 con 0x7fcb8009cb30 2024-08-16T02:37:11.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.477+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcb80076690 con 0x7fcb8009cb30 2024-08-16T02:37:11.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.479+0000 7fcb67fff640 1 -- 172.21.15.116:0/551634527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+96288 (crc 0 0 0) 0x7fcb70011e60 con 0x7fcb8009cb30 2024-08-16T02:37:11.481 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 msgr2=0x7fcb6c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 0x7fcb6c040220 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 msgr2=0x7fcb801207a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 crc :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 shutdown_connections 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fcb6c03dd10 0x7fcb6c040220 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 --2- 172.21.15.116:0/551634527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb8009cb30 0x7fcb801207a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 >> 172.21.15.116:0/551634527 conn(0x7fcb800a09c0 msgr2=0x7fcb800a1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 shutdown_connections 2024-08-16T02:37:11.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.482+0000 7fcb85b52640 1 -- 172.21.15.116:0/551634527 wait complete. 2024-08-16T02:37:11.494 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":3181043282},{"type":"v1","addr":"172.21.15.152:6801","nonce":3181043282}]},"active_addr":"172.21.15.152:6801/3181043282","active_change":"2024-08-16T02:37:01.655715+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":1348773374}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":895007322}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":2440258117}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":3456662359}]}]} 2024-08-16T02:37:11.497 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-16T02:37:11.497 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-16T02:37:11.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:37:11.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.600+0000 7fad074ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.604+0000 7fad074ad640 1 Processor -- start 2024-08-16T02:37:11.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.605+0000 7fad074ad640 1 -- start start 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad074ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad00135150 0x7fad00137570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad074ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0009ca30 con 0x7fad001322e0 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad074ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fad0009e6b0 con 0x7fad00135150 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad00135150 0x7fad00137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad00135150 0x7fad00137570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47434/0 (socket says 172.21.15.116:47434) 2024-08-16T02:37:11.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 -- 172.21.15.116:0/2695842455 learned_addr learned my addr 172.21.15.116:0/2695842455 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad05249640 1 --1- 172.21.15.116:0/2695842455 >> v1:172.21.15.116:6789/0 conn(0x7fad001322e0 0x7fad001326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44908/0 (socket says 172.21.15.116:44908) 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 -- 172.21.15.116:0/2695842455 >> v1:172.21.15.116:6789/0 conn(0x7fad001322e0 legacy=0x7fad001326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7facf7fff640 1 -- 172.21.15.116:0/2695842455 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1372865586 0 0) 0x7fad0009ca30 con 0x7fad001322e0 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 -- 172.21.15.116:0/2695842455 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0009e3a0 con 0x7fad00135150 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7fad04a48640 1 --2- 172.21.15.116:0/2695842455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00135150 0x7fad00137570 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158ef87ae47b1c52 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7facf7fff640 1 -- 172.21.15.116:0/2695842455 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7facfc009e50 con 0x7fad00135150 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7facf7fff640 1 -- 172.21.15.116:0/2695842455 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7facfc00e230 con 0x7fad00135150 2024-08-16T02:37:11.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.606+0000 7facf7fff640 1 -- 172.21.15.116:0/2695842455 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7facfc00e4d0 con 0x7fad00135150 2024-08-16T02:37:11.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 -- 172.21.15.116:0/2695842455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00135150 msgr2=0x7fad00137570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 --2- 172.21.15.116:0/2695842455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00135150 0x7fad00137570 crc :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 -- 172.21.15.116:0/2695842455 shutdown_connections 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 --2- 172.21.15.116:0/2695842455 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad00135150 0x7fad00137570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 -- 172.21.15.116:0/2695842455 >> 172.21.15.116:0/2695842455 conn(0x7fad0009f1d0 msgr2=0x7fad0009f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 -- 172.21.15.116:0/2695842455 shutdown_connections 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.607+0000 7fad074ad640 1 -- 172.21.15.116:0/2695842455 wait complete. 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.608+0000 7fad074ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.609+0000 7fad074ad640 1 Processor -- start 2024-08-16T02:37:11.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.609+0000 7fad074ad640 1 -- start start 2024-08-16T02:37:11.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad074ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad074ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad001220a0 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad05249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad05249640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47450/0 (socket says 172.21.15.116:47450) 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad05249640 1 -- 172.21.15.116:0/566275192 learned_addr learned my addr 172.21.15.116:0/566275192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad05249640 1 -- 172.21.15.116:0/566275192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad00131760 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad05249640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7facf0004b80 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7facf0004ce0 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad00131d80 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.610+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7facf0007120 con 0x7fad001322e0 2024-08-16T02:37:11.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.611+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad00123990 con 0x7fad001322e0 2024-08-16T02:37:11.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.611+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7facf00072d0 con 0x7fad001322e0 2024-08-16T02:37:11.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.612+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad001326f0 con 0x7fad001322e0 2024-08-16T02:37:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.612+0000 7facf5ffb640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 0x7face4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.612+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7facf004e860 con 0x7fad001322e0 2024-08-16T02:37:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.612+0000 7fad04a48640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 0x7face4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.613+0000 7fad04a48640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 0x7face4040220 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.617+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7facf000adb0 con 0x7fad001322e0 2024-08-16T02:37:11.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.803+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad0006bf10 con 0x7fad001322e0 2024-08-16T02:37:11.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.804+0000 7facf5ffb640 1 -- 172.21.15.116:0/566275192 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8891 (crc 0 0 0) 0x7facf000d150 con 0x7fad001322e0 2024-08-16T02:37:11.805 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:11.805 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:37:10.749181+0000","last_up_change":"2024-08-16T02:37:03.718029+0000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:37:05.689624+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:37:07.533370+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":295590560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":295590560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":295590560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6805","nonce":295590560}]},"public_addr":"172.21.15.116:6804/295590560","cluster_addr":"172.21.15.116:6803/295590560","heartbeat_back_addr":"172.21.15.116:6806/295590560","heartbeat_front_addr":"172.21.15.116:6805/295590560","state":["exists","up"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2086482905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6801","nonce":2086482905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":2086482905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2086482905}]},"public_addr":"172.21.15.116:6802/2086482905","cluster_addr":"172.21.15.116:6801/2086482905","heartbeat_back_addr":"172.21.15.116:6807/2086482905","heartbeat_front_addr":"172.21.15.116:6808/2086482905","state":["exists","up"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1500775361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":1500775361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1500775361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":1500775361}]},"public_addr":"172.21.15.152:6820/1500775361","cluster_addr":"172.21.15.152:6821/1500775361","heartbeat_back_addr":"172.21.15.152:6806/1500775361","heartbeat_front_addr":"172.21.15.152:6807/1500775361","state":["exists","up"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2721550083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2721550083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2721550083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2721550083}]},"public_addr":"172.21.15.152:6804/2721550083","cluster_addr":"172.21.15.152:6803/2721550083","heartbeat_back_addr":"172.21.15.152:6805/2721550083","heartbeat_front_addr":"172.21.15.152:6802/2721550083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:37:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 msgr2=0x7face4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 0x7face4040220 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 msgr2=0x7fad00120bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 crc :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 shutdown_connections 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7face403dd10 0x7face4040220 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 --2- 172.21.15.116:0/566275192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad001322e0 0x7fad00120bb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 >> 172.21.15.116:0/566275192 conn(0x7fad0009f1d0 msgr2=0x7fad00132f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 shutdown_connections 2024-08-16T02:37:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.806+0000 7fad074ad640 1 -- 172.21.15.116:0/566275192 wait complete. 2024-08-16T02:37:11.818 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-16T02:37:11.818 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:37:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.922+0000 7f3a370ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.927+0000 7f3a370ad640 1 Processor -- start 2024-08-16T02:37:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.927+0000 7f3a370ad640 1 -- start start 2024-08-16T02:37:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a370ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a300a0950 0x7f3a3013a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a370ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a300a14c0 con 0x7f3a3009fe30 2024-08-16T02:37:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a370ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3a3006f380 con 0x7f3a300a0950 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a300a0950 0x7f3a3013a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a34e49640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a3009fe30 0x7f3a300a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44924/0 (socket says 172.21.15.116:44924) 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a300a0950 0x7f3a3013a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47458/0 (socket says 172.21.15.116:47458) 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a34e49640 1 -- 172.21.15.116:0/1114080396 learned_addr learned my addr 172.21.15.116:0/1114080396 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2ffff640 1 -- 172.21.15.116:0/1114080396 >> v1:172.21.15.116:6789/0 conn(0x7f3a3009fe30 legacy=0x7f3a300a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2ffff640 1 -- 172.21.15.116:0/1114080396 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a300a2130 con 0x7f3a300a0950 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2f7fe640 1 -- 172.21.15.116:0/1114080396 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292410291 0 0) 0x7f3a300a14c0 con 0x7f3a3009fe30 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.928+0000 7f3a2ffff640 1 --2- 172.21.15.116:0/1114080396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a300a0950 0x7f3a3013a7e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100a3cf8e0295d15 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a2f7fe640 1 -- 172.21.15.116:0/1114080396 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a24009d90 con 0x7f3a300a0950 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a2f7fe640 1 -- 172.21.15.116:0/1114080396 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3a2400e230 con 0x7f3a300a0950 2024-08-16T02:37:11.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a2f7fe640 1 -- 172.21.15.116:0/1114080396 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a2400e4d0 con 0x7f3a300a0950 2024-08-16T02:37:11.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 -- 172.21.15.116:0/1114080396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a300a0950 msgr2=0x7f3a3013a7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 --2- 172.21.15.116:0/1114080396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a300a0950 0x7f3a3013a7e0 crc :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 -- 172.21.15.116:0/1114080396 shutdown_connections 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 --2- 172.21.15.116:0/1114080396 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a300a0950 0x7f3a3013a7e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 -- 172.21.15.116:0/1114080396 >> 172.21.15.116:0/1114080396 conn(0x7f3a3006caf0 msgr2=0x7f3a3006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 -- 172.21.15.116:0/1114080396 shutdown_connections 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.929+0000 7f3a370ad640 1 -- 172.21.15.116:0/1114080396 wait complete. 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.930+0000 7f3a370ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.931+0000 7f3a370ad640 1 Processor -- start 2024-08-16T02:37:11.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.931+0000 7f3a370ad640 1 -- start start 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a370ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a370ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a3013a420 con 0x7f3a3009fe30 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a34e49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a34e49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47468/0 (socket says 172.21.15.116:47468) 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a34e49640 1 -- 172.21.15.116:0/79710895 learned_addr learned my addr 172.21.15.116:0/79710895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.932+0000 7f3a34e49640 1 -- 172.21.15.116:0/79710895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a30125d60 con 0x7f3a3009fe30 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a34e49640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a20004bc0 con 0x7f3a3009fe30 2024-08-16T02:37:11.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3a20004d20 con 0x7f3a3009fe30 2024-08-16T02:37:11.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a301265c0 con 0x7f3a3009fe30 2024-08-16T02:37:11.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3a20007120 con 0x7f3a3009fe30 2024-08-16T02:37:11.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.933+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a3006fa40 con 0x7f3a3009fe30 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.934+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a30128a10 con 0x7f3a3009fe30 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.934+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f3a200072d0 con 0x7f3a3009fe30 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.934+0000 7f3a2d7fa640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 0x7f3a1c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.934+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f3a2004f010 con 0x7f3a3009fe30 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.934+0000 7f3a2ffff640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 0x7f3a1c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:11.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.935+0000 7f3a2ffff640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 0x7f3a1c040220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:11.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:11.939+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f3a20006290 con 0x7f3a3009fe30 2024-08-16T02:37:12.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.124+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a300a0240 con 0x7f3a3009fe30 2024-08-16T02:37:12.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.125+0000 7f3a2d7fa640 1 -- 172.21.15.116:0/79710895 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+8891 (crc 0 0 0) 0x7f3a200060b0 con 0x7f3a3009fe30 2024-08-16T02:37:12.127 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:12.127 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":15,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:37:10.749181+0000","last_up_change":"2024-08-16T02:37:03.718029+0000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:37:05.689624+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:37:07.533370+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":295590560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":295590560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":295590560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6805","nonce":295590560}]},"public_addr":"172.21.15.116:6804/295590560","cluster_addr":"172.21.15.116:6803/295590560","heartbeat_back_addr":"172.21.15.116:6806/295590560","heartbeat_front_addr":"172.21.15.116:6805/295590560","state":["exists","up"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2086482905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6801","nonce":2086482905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":2086482905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2086482905}]},"public_addr":"172.21.15.116:6802/2086482905","cluster_addr":"172.21.15.116:6801/2086482905","heartbeat_back_addr":"172.21.15.116:6807/2086482905","heartbeat_front_addr":"172.21.15.116:6808/2086482905","state":["exists","up"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1500775361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":1500775361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1500775361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":1500775361}]},"public_addr":"172.21.15.152:6820/1500775361","cluster_addr":"172.21.15.152:6821/1500775361","heartbeat_back_addr":"172.21.15.152:6806/1500775361","heartbeat_front_addr":"172.21.15.152:6807/1500775361","state":["exists","up"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2721550083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2721550083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2721550083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2721550083}]},"public_addr":"172.21.15.152:6804/2721550083","cluster_addr":"172.21.15.152:6803/2721550083","heartbeat_back_addr":"172.21.15.152:6805/2721550083","heartbeat_front_addr":"172.21.15.152:6802/2721550083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.127+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 msgr2=0x7f3a1c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.127+0000 7f3a370ad640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 0x7f3a1c040220 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.127+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 msgr2=0x7f3a301c9c50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 crc :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 shutdown_connections 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3a1c03dd10 0x7f3a1c040220 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 --2- 172.21.15.116:0/79710895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a3009fe30 0x7f3a301c9c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 >> 172.21.15.116:0/79710895 conn(0x7f3a3006caf0 msgr2=0x7f3a30072dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 shutdown_connections 2024-08-16T02:37:12.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.128+0000 7f3a370ad640 1 -- 172.21.15.116:0/79710895 wait complete. 2024-08-16T02:37:12.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-16T02:37:12.141 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-16T02:37:12.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-16T02:37:12.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-16T02:37:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.254+0000 7f2de3824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.254+0000 7ff473712640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.254+0000 7fb86fa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.254+0000 7fa630d92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7ff473712640 1 Processor -- start 2024-08-16T02:37:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7f2de3824640 1 Processor -- start 2024-08-16T02:37:12.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7fb86fa2d640 1 Processor -- start 2024-08-16T02:37:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7ff473712640 1 -- start start 2024-08-16T02:37:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7fb86fa2d640 1 -- start start 2024-08-16T02:37:12.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7f2de3824640 1 -- start start 2024-08-16T02:37:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7fa630d92640 1 Processor -- start 2024-08-16T02:37:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.260+0000 7fa630d92640 1 -- start start 2024-08-16T02:37:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86fa2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb868131af0 0x7fb868131f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff473712640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff46c136340 0x7ff46c138760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff473712640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff46c133010 con 0x7ff46c1333d0 2024-08-16T02:37:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff473712640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff46c0a4db0 con 0x7ff46c136340 2024-08-16T02:37:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de3824640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2ddc138490 0x7f2ddc13a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de3824640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ddc0a4f10 con 0x7f2ddc135490 2024-08-16T02:37:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de3824640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2ddc135fb0 con 0x7f2ddc138490 2024-08-16T02:37:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86fa2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb86806ffd0 con 0x7fb86809e2c0 2024-08-16T02:37:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86fa2d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb8681316a0 con 0x7fb868131af0 2024-08-16T02:37:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de15c0640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2ddc135490 0x7f2ddc1358a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44956/0 (socket says 172.21.15.116:44956) 2024-08-16T02:37:12.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de15c0640 1 -- 172.21.15.116:0/3706231001 learned_addr learned my addr 172.21.15.116:0/3706231001 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86cfc8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb86809e2c0 0x7fb86809e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44948/0 (socket says 172.21.15.116:44948) 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86d7c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb868131af0 0x7fb868131f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7fb86cfc8640 1 -- 172.21.15.116:0/2339037180 learned_addr learned my addr 172.21.15.116:0/2339037180 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff470cad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff46c136340 0x7ff46c138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff470cad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff46c136340 0x7ff46c138760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47494/0 (socket says 172.21.15.116:47494) 2024-08-16T02:37:12.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7ff470cad640 1 -- 172.21.15.116:0/1814361136 learned_addr learned my addr 172.21.15.116:0/1814361136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.261+0000 7f2de0dbf640 1 --2- 172.21.15.116:0/3706231001 >> v2:172.21.15.116:3300/0 conn(0x7f2ddc138490 0x7f2ddc13a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa630d92640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa62c09c540 0x7fa62c137480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa630d92640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c09e5e0 con 0x7fa62c09ba20 2024-08-16T02:37:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa630d92640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa62c09ef10 con 0x7fa62c09c540 2024-08-16T02:37:12.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7ff470cad640 1 -- 172.21.15.116:0/1814361136 >> v1:172.21.15.116:6789/0 conn(0x7ff46c1333d0 legacy=0x7ff46c1337e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa62c09c540 0x7fa62c137480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7ff470cad640 1 -- 172.21.15.116:0/1814361136 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c133ef0 con 0x7ff46c136340 2024-08-16T02:37:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa62c09c540 0x7fa62c137480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47516/0 (socket says 172.21.15.116:47516) 2024-08-16T02:37:12.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 -- 172.21.15.116:0/3757904040 learned_addr learned my addr 172.21.15.116:0/3757904040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fb86d7c9640 1 -- 172.21.15.116:0/2339037180 >> v1:172.21.15.116:6789/0 conn(0x7fb86809e2c0 legacy=0x7fb86809e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fb86d7c9640 1 -- 172.21.15.116:0/2339037180 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86809fea0 con 0x7fb868131af0 2024-08-16T02:37:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7f2de0dbf640 1 -- 172.21.15.116:0/3706231001 >> v1:172.21.15.116:6789/0 conn(0x7f2ddc135490 legacy=0x7f2ddc1358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7f2dd3fff640 1 -- 172.21.15.116:0/3706231001 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2688182781 0 0) 0x7f2ddc0a4f10 con 0x7f2ddc135490 2024-08-16T02:37:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7f2de0dbf640 1 -- 172.21.15.116:0/3706231001 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc072b20 con 0x7f2ddc138490 2024-08-16T02:37:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 -- 172.21.15.116:0/3757904040 >> v1:172.21.15.116:6789/0 conn(0x7fa62c09ba20 legacy=0x7fa62c09be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 -- 172.21.15.116:0/3757904040 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c09ece0 con 0x7fa62c09c540 2024-08-16T02:37:12.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fb86d7c9640 1 --2- 172.21.15.116:0/2339037180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb868131af0 0x7fb868131f00 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b4df458e893e30a server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7ff470cad640 1 --2- 172.21.15.116:0/1814361136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c136340 0x7ff46c138760 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b32a9c809a1540 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7fa623fff640 1 --2- 172.21.15.116:0/3757904040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09c540 0x7fa62c137480 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4d182775a908e7 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7f2de0dbf640 1 --2- 172.21.15.116:0/3706231001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc138490 0x7f2ddc13a8b0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b5877a22cd3d24 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.262+0000 7ff463fff640 1 -- 172.21.15.116:0/1814361136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff45c009e50 con 0x7ff46c136340 2024-08-16T02:37:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7ff463fff640 1 -- 172.21.15.116:0/1814361136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff45c00e230 con 0x7ff46c136340 2024-08-16T02:37:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7ff463fff640 1 -- 172.21.15.116:0/1814361136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff45c00e4d0 con 0x7ff46c136340 2024-08-16T02:37:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fb85ffff640 1 -- 172.21.15.116:0/2339037180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb86400ace0 con 0x7fb868131af0 2024-08-16T02:37:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fa629d9b640 1 -- 172.21.15.116:0/3757904040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa614009e50 con 0x7fa62c09c540 2024-08-16T02:37:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fa629d9b640 1 -- 172.21.15.116:0/3757904040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa614010440 con 0x7fa62c09c540 2024-08-16T02:37:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fa629d9b640 1 -- 172.21.15.116:0/3757904040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa6140106e0 con 0x7fa62c09c540 2024-08-16T02:37:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fb85ffff640 1 -- 172.21.15.116:0/2339037180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb86400ae40 con 0x7fb868131af0 2024-08-16T02:37:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7fb85ffff640 1 -- 172.21.15.116:0/2339037180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb86400f5a0 con 0x7fb868131af0 2024-08-16T02:37:12.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7f2dd3fff640 1 -- 172.21.15.116:0/3706231001 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2dcc009d90 con 0x7f2ddc138490 2024-08-16T02:37:12.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7f2dd3fff640 1 -- 172.21.15.116:0/3706231001 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2dcc00e230 con 0x7f2ddc138490 2024-08-16T02:37:12.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.263+0000 7f2dd3fff640 1 -- 172.21.15.116:0/3706231001 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2dcc00e4d0 con 0x7f2ddc138490 2024-08-16T02:37:12.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 -- 172.21.15.116:0/1814361136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c136340 msgr2=0x7ff46c138760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 --2- 172.21.15.116:0/1814361136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c136340 0x7ff46c138760 crc :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 -- 172.21.15.116:0/3757904040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09c540 msgr2=0x7fa62c137480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 --2- 172.21.15.116:0/3757904040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09c540 0x7fa62c137480 crc :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 -- 172.21.15.116:0/3757904040 shutdown_connections 2024-08-16T02:37:12.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 -- 172.21.15.116:0/2339037180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb868131af0 msgr2=0x7fb868131f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/2339037180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb868131af0 0x7fb868131f00 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 --2- 172.21.15.116:0/3757904040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09c540 0x7fa62c137480 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 -- 172.21.15.116:0/3757904040 >> 172.21.15.116:0/3757904040 conn(0x7fa62c06caf0 msgr2=0x7fa62c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 -- 172.21.15.116:0/2339037180 shutdown_connections 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/2339037180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb868131af0 0x7fb868131f00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 -- 172.21.15.116:0/2339037180 >> 172.21.15.116:0/2339037180 conn(0x7fb8680a1550 msgr2=0x7fb8680a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 -- 172.21.15.116:0/1814361136 shutdown_connections 2024-08-16T02:37:12.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 --2- 172.21.15.116:0/1814361136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c136340 0x7ff46c138760 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 -- 172.21.15.116:0/3706231001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc138490 msgr2=0x7f2ddc13a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 --2- 172.21.15.116:0/3706231001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc138490 0x7f2ddc13a8b0 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 -- 172.21.15.116:0/1814361136 >> 172.21.15.116:0/1814361136 conn(0x7ff46c0a2270 msgr2=0x7ff46c0a2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 -- 172.21.15.116:0/3757904040 shutdown_connections 2024-08-16T02:37:12.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 -- 172.21.15.116:0/3706231001 shutdown_connections 2024-08-16T02:37:12.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 --2- 172.21.15.116:0/3706231001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc138490 0x7f2ddc13a8b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 -- 172.21.15.116:0/3706231001 >> 172.21.15.116:0/3706231001 conn(0x7f2ddc0a5880 msgr2=0x7f2ddc0a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 1 -- 172.21.15.116:0/3757904040 wait complete. 2024-08-16T02:37:12.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 -- 172.21.15.116:0/2339037180 shutdown_connections 2024-08-16T02:37:12.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 -- 172.21.15.116:0/1814361136 shutdown_connections 2024-08-16T02:37:12.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 1 -- 172.21.15.116:0/1814361136 wait complete. 2024-08-16T02:37:12.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 1 -- 172.21.15.116:0/2339037180 wait complete. 2024-08-16T02:37:12.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 -- 172.21.15.116:0/3706231001 shutdown_connections 2024-08-16T02:37:12.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 1 -- 172.21.15.116:0/3706231001 wait complete. 2024-08-16T02:37:12.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fb86fa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7fa630d92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7f2de3824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.265+0000 7fb86fa2d640 1 Processor -- start 2024-08-16T02:37:12.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.264+0000 7ff473712640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.265+0000 7ff473712640 1 Processor -- start 2024-08-16T02:37:12.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7fb86fa2d640 1 -- start start 2024-08-16T02:37:12.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7f2de3824640 1 Processor -- start 2024-08-16T02:37:12.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7fa630d92640 1 Processor -- start 2024-08-16T02:37:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7f2de3824640 1 -- start start 2024-08-16T02:37:12.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7fa630d92640 1 -- start start 2024-08-16T02:37:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.266+0000 7ff473712640 1 -- start start 2024-08-16T02:37:12.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86fa2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7ff473712640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7ff473712640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff46c11f470 con 0x7ff46c1333d0 2024-08-16T02:37:12.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86fa2d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb86812efe0 con 0x7fb86809e2c0 2024-08-16T02:37:12.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86d7c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7ff4714ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86d7c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47540/0 (socket says 172.21.15.116:47540) 2024-08-16T02:37:12.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7f2de3824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86d7c9640 1 -- 172.21.15.116:0/3668768337 learned_addr learned my addr 172.21.15.116:0/3668768337 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7f2de3824640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2ddc120c50 con 0x7f2ddc135490 2024-08-16T02:37:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7ff4714ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47530/0 (socket says 172.21.15.116:47530) 2024-08-16T02:37:12.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7ff4714ae640 1 -- 172.21.15.116:0/2506010327 learned_addr learned my addr 172.21.15.116:0/2506010327 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7f2de15c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fa630d92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fa630d92640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c121550 con 0x7fa62c09ba20 2024-08-16T02:37:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7f2de15c0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47546/0 (socket says 172.21.15.116:47546) 2024-08-16T02:37:12.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7f2de15c0640 1 -- 172.21.15.116:0/801031911 learned_addr learned my addr 172.21.15.116:0/801031911 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fa62a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47552/0 (socket says 172.21.15.116:47552) 2024-08-16T02:37:12.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fa62a59c640 1 -- 172.21.15.116:0/2981963804 learned_addr learned my addr 172.21.15.116:0/2981963804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86d7c9640 1 -- 172.21.15.116:0/3668768337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb868125fa0 con 0x7fb86809e2c0 2024-08-16T02:37:12.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.267+0000 7fb86d7c9640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fa62a59c640 1 -- 172.21.15.116:0/2981963804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c120d10 con 0x7fa62c09ba20 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb86400e400 con 0x7fb86809e2c0 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fb86400e560 con 0x7fb86809e2c0 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fb86400e800 con 0x7fb86809e2c0 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86812ecb0 con 0x7fb86809e2c0 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2de15c0640 1 -- 172.21.15.116:0/801031911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ddc123000 con 0x7f2ddc135490 2024-08-16T02:37:12.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7ff4714ae640 1 -- 172.21.15.116:0/2506010327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff46c11e650 con 0x7ff46c1333d0 2024-08-16T02:37:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fa62a59c640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681312e0 con 0x7fb86809e2c0 2024-08-16T02:37:12.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2de15c0640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7ff4714ae640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2dd8004b80 con 0x7f2ddc135490 2024-08-16T02:37:12.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f2dd8004ce0 con 0x7f2ddc135490 2024-08-16T02:37:12.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f2dd8007120 con 0x7f2ddc135490 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ddc122650 con 0x7f2ddc135490 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa61c00c810 con 0x7fa62c09ba20 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.268+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa61c00c970 con 0x7fa62c09ba20 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff46800c810 con 0x7ff46c1333d0 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7ff46800c970 con 0x7ff46c1333d0 2024-08-16T02:37:12.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7ff46800cc10 con 0x7ff46c1333d0 2024-08-16T02:37:12.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff46c073760 con 0x7ff46c1333d0 2024-08-16T02:37:12.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ddc123ff0 con 0x7f2ddc135490 2024-08-16T02:37:12.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb868077770 con 0x7fb86809e2c0 2024-08-16T02:37:12.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff46c1c21d0 con 0x7ff46c1333d0 2024-08-16T02:37:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fb86400e9b0 con 0x7fb86809e2c0 2024-08-16T02:37:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa61c00cc10 con 0x7fa62c09ba20 2024-08-16T02:37:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c132e20 con 0x7fa62c09ba20 2024-08-16T02:37:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c1223e0 con 0x7fa62c09ba20 2024-08-16T02:37:12.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.269+0000 7fb85dffb640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 0x7fb84c040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fb864014040 con 0x7fb86809e2c0 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fb85dffb640 1 --2- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 0x7fb84c045210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 --> v2:172.21.15.152:6820/1500775361 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb84c0458a0 con 0x7fb84c042d40 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7fb864015e90 con 0x7fb86809e2c0 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2ddc120fe0 con 0x7f2ddc135490 2024-08-16T02:37:12.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f2dd80072d0 con 0x7f2ddc135490 2024-08-16T02:37:12.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.270+0000 7fb86cfc8640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 0x7fb84c040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2dd1ffb640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 0x7f2dc0040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f2dd804e950 con 0x7f2ddc135490 2024-08-16T02:37:12.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa62c1239a0 con 0x7fa62c09ba20 2024-08-16T02:37:12.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff44c000f80 con 0x7ff46c1333d0 2024-08-16T02:37:12.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7ff468011310 con 0x7ff46c1333d0 2024-08-16T02:37:12.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2dd1ffb640 1 --2- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 0x7f2dc00451e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 --> v2:172.21.15.152:6804/2721550083 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2dc0045870 con 0x7f2dc0042d10 2024-08-16T02:37:12.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7f2dd804ec60 con 0x7f2ddc135490 2024-08-16T02:37:12.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7f2de0dbf640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 0x7f2dc0040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 0x7ff4500402c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7ff46804ee40 con 0x7ff46c1333d0 2024-08-16T02:37:12.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fa61c011310 con 0x7fa62c09ba20 2024-08-16T02:37:12.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7fb86cfc8640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 0x7fb84c040250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 --2- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 0x7ff450045300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 --> v2:172.21.15.116:6804/295590560 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff450045990 con 0x7ff450042e30 2024-08-16T02:37:12.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7ff46804f180 con 0x7ff46c1333d0 2024-08-16T02:37:12.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7fb86dfca640 1 --2- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 0x7fb84c045210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:12.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.271+0000 7ff470cad640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 0x7ff4500402c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa622ffd640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 0x7fa60c040190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fa61c04edd0 con 0x7fa62c09ba20 2024-08-16T02:37:12.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7f2de0dbf640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 0x7f2dc0040220 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7f2de1dc1640 1 --2- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 0x7f2dc00451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:12.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa622ffd640 1 --2- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 0x7fa60c044fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 --> v2:172.21.15.116:6802/2086482905 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa60c045670 con 0x7fa60c042b10 2024-08-16T02:37:12.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa623fff640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 0x7fa60c040190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (crc 0 0 0) 0x7fa61c00cdc0 con 0x7fa62c09ba20 2024-08-16T02:37:12.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.272+0000 7ff470cad640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 0x7ff4500402c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.273+0000 7ff471caf640 1 --2- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 0x7ff450045300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:12.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.273+0000 7fa623fff640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 0x7fa60c040190 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.273+0000 7fa62ad9d640 1 --2- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 0x7fa60c044fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:37:12.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.275+0000 7f2de1dc1640 1 --2- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 0x7f2dc00451e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.276+0000 7fb86dfca640 1 --2- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 0x7fb84c045210 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.276+0000 7ff471caf640 1 --2- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 0x7ff450045300 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.276+0000 7fa62ad9d640 1 --2- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 0x7fa60c044fe0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.277+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== osd.3 v2:172.21.15.152:6804/2721550083 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7f2dc0045870 con 0x7f2dc0042d10 2024-08-16T02:37:12.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.278+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== osd.1 v2:172.21.15.116:6802/2086482905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fa60c045670 con 0x7fa60c042b10 2024-08-16T02:37:12.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.278+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== osd.2 v2:172.21.15.152:6820/1500775361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fb84c0458a0 con 0x7fb84c042d40 2024-08-16T02:37:12.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.279+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== osd.0 v2:172.21.15.116:6804/295590560 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7ff450045990 con 0x7ff450042e30 2024-08-16T02:37:12.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.281+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 --> v2:172.21.15.152:6804/2721550083 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2ddc138230 con 0x7f2dc0042d10 2024-08-16T02:37:12.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.282+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 --> v2:172.21.15.152:6820/1500775361 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb86806fb40 con 0x7fb84c042d40 2024-08-16T02:37:12.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2dd1ffb640 1 -- 172.21.15.116:0/801031911 <== osd.3 v2:172.21.15.152:6804/2721550083 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2ddc138230 con 0x7f2dc0042d10 2024-08-16T02:37:12.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 msgr2=0x7f2dc00451e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 0x7f2dc00451e0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 msgr2=0x7f2dc0040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 0x7f2dc0040220 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 msgr2=0x7f2ddc1c9d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 crc :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 shutdown_connections 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> v2:172.21.15.152:6804/2721550083 conn(0x7f2dc0042d10 0x7f2dc00451e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f2dc003dd10 0x7f2dc0040220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 --2- 172.21.15.116:0/801031911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2ddc135490 0x7f2ddc1c9d50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 >> 172.21.15.116:0/801031911 conn(0x7f2ddc0a5880 msgr2=0x7f2ddc070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 shutdown_connections 2024-08-16T02:37:12.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.283+0000 7f2de3824640 1 -- 172.21.15.116:0/801031911 wait complete. 2024-08-16T02:37:12.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb85dffb640 1 -- 172.21.15.116:0/3668768337 <== osd.2 v2:172.21.15.152:6820/1500775361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb86000a820 con 0x7fb84c042d40 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 msgr2=0x7fb84c045210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 0x7fb84c045210 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 msgr2=0x7fb84c040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 0x7fb84c040250 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 msgr2=0x7fb868121fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 crc :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 shutdown_connections 2024-08-16T02:37:12.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> v2:172.21.15.152:6820/1500775361 conn(0x7fb84c042d40 0x7fb84c045210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fb84c03dd40 0x7fb84c040250 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 --2- 172.21.15.116:0/3668768337 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb86809e2c0 0x7fb868121fc0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 >> 172.21.15.116:0/3668768337 conn(0x7fb8680a1550 msgr2=0x7fb86809cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 shutdown_connections 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.284+0000 7fb86fa2d640 1 -- 172.21.15.116:0/3668768337 wait complete. 2024-08-16T02:37:12.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.285+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 --> v2:172.21.15.116:6802/2086482905 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa62c09c790 con 0x7fa60c042b10 2024-08-16T02:37:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.285+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 --> v2:172.21.15.116:6804/295590560 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff44c002c30 con 0x7ff450042e30 2024-08-16T02:37:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.287+0000 7ff461ffb640 1 -- 172.21.15.116:0/2506010327 <== osd.0 v2:172.21.15.116:6804/295590560 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff44c002c30 con 0x7ff450042e30 2024-08-16T02:37:12.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.287+0000 7fa622ffd640 1 -- 172.21.15.116:0/2981963804 <== osd.1 v2:172.21.15.116:6802/2086482905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa62c09c790 con 0x7fa60c042b10 2024-08-16T02:37:12.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 msgr2=0x7fa60c044fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 0x7fa60c044fe0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 msgr2=0x7ff450045300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 0x7ff450045300 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 msgr2=0x7ff4500402c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 0x7ff4500402c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 msgr2=0x7ff46c11cbd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 crc :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 shutdown_connections 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> v2:172.21.15.116:6804/295590560 conn(0x7ff450042e30 0x7ff450045300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7ff45003ddb0 0x7ff4500402c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 msgr2=0x7fa60c040190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.288+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 0x7fa60c040190 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7ff473712640 1 --2- 172.21.15.116:0/2506010327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff46c1333d0 0x7ff46c11cbd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 >> 172.21.15.116:0/2506010327 conn(0x7ff46c0a2270 msgr2=0x7ff46c0a0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 msgr2=0x7fa62c11fad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 crc :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 shutdown_connections 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> v2:172.21.15.116:6802/2086482905 conn(0x7fa60c042b10 0x7fa60c044fe0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa60c03dc80 0x7fa60c040190 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 --2- 172.21.15.116:0/2981963804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa62c09ba20 0x7fa62c11fad0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 >> 172.21.15.116:0/2981963804 conn(0x7fa62c06caf0 msgr2=0x7fa62c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 shutdown_connections 2024-08-16T02:37:12.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7fa630d92640 1 -- 172.21.15.116:0/2981963804 wait complete. 2024-08-16T02:37:12.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 shutdown_connections 2024-08-16T02:37:12.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.289+0000 7ff473712640 1 -- 172.21.15.116:0/2506010327 wait complete. 2024-08-16T02:37:12.328 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-16T02:37:12.329 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.329 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-16T02:37:12.330 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-16T02:37:12.331 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-16T02:37:12.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.482+0000 7fd57cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.482+0000 7f8bb746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.482+0000 7fa45c464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.481+0000 7fc8f1464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7f8bb746d640 1 Processor -- start 2024-08-16T02:37:12.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7fc8f1464640 1 Processor -- start 2024-08-16T02:37:12.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7fd57cc6d640 1 Processor -- start 2024-08-16T02:37:12.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7fd57cc6d640 1 -- start start 2024-08-16T02:37:12.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7fc8f1464640 1 -- start start 2024-08-16T02:37:12.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.487+0000 7f8bb746d640 1 -- start start 2024-08-16T02:37:12.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.488+0000 7fa45c464640 1 Processor -- start 2024-08-16T02:37:12.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.488+0000 7fa45c464640 1 -- start start 2024-08-16T02:37:12.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb746d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8bb009fb90 0x7f8bb0131c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb746d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bb01317c0 con 0x7f8bb0132150 2024-08-16T02:37:12.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb746d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8bb0070470 con 0x7f8bb009fb90 2024-08-16T02:37:12.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd57cc6d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd5781384c0 0x7fd57813a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd57cc6d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd57806fba0 con 0x7fd578135570 2024-08-16T02:37:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb4a08640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8bb0132150 0x7f8bb009e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44970/0 (socket says 172.21.15.116:44970) 2024-08-16T02:37:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb4a08640 1 -- 172.21.15.116:0/1032090236 learned_addr learned my addr 172.21.15.116:0/1032090236 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:12.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8bb5209640 1 --2- 172.21.15.116:0/1032090236 >> v2:172.21.15.116:3300/0 conn(0x7f8bb009fb90 0x7f8bb0131c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd57cc6d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd5780a2f30 con 0x7fd5781384c0 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd57659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd578135570 0x7fd578135980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44980/0 (socket says 172.21.15.116:44980) 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd57659c640 1 -- 172.21.15.116:0/277747350 learned_addr learned my addr 172.21.15.116:0/277747350 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd575d9b640 1 --2- 172.21.15.116:0/277747350 >> v2:172.21.15.116:3300/0 conn(0x7fd5781384c0 0x7fd57813a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd575d9b640 1 -- 172.21.15.116:0/277747350 >> v1:172.21.15.116:6789/0 conn(0x7fd578135570 legacy=0x7fd578135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd575d9b640 1 -- 172.21.15.116:0/277747350 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5780a3920 con 0x7fd5781384c0 2024-08-16T02:37:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fc8f1464640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 0x7fc8ec09ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8ba7fff640 1 -- 172.21.15.116:0/1032090236 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423857808 0 0) 0x7f8bb01317c0 con 0x7f8bb0132150 2024-08-16T02:37:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7f8ba7fff640 1 -- 172.21.15.116:0/1032090236 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b9c0034a0 con 0x7f8bb0132150 2024-08-16T02:37:12.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.489+0000 7fd575d9b640 1 --2- 172.21.15.116:0/277747350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5781384c0 0x7fd57813a8e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9afe33b261f1b3a0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7f8bb5209640 1 -- 172.21.15.116:0/1032090236 >> v1:172.21.15.116:6789/0 conn(0x7f8bb0132150 legacy=0x7f8bb009e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8eaffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 0x7fc8ec09ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7f8bb5209640 1 -- 172.21.15.116:0/1032090236 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb006f8d0 con 0x7f8bb009fb90 2024-08-16T02:37:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8eaffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 0x7fc8ec09ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47572/0 (socket says 172.21.15.116:47572) 2024-08-16T02:37:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8eaffd640 1 -- 172.21.15.116:0/1466599527 learned_addr learned my addr 172.21.15.116:0/1466599527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ec09e750 con 0x7fc8ec09bf60 2024-08-16T02:37:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fd57559a640 1 -- 172.21.15.116:0/277747350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd56c009d90 con 0x7fd5781384c0 2024-08-16T02:37:12.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fd57559a640 1 -- 172.21.15.116:0/277747350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd56c00e230 con 0x7fd5781384c0 2024-08-16T02:37:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc8ec09e9f0 con 0x7fc8ec09d5f0 2024-08-16T02:37:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa45c464640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa454132cc0 0x7fa4541376d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa45c464640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa45409dc90 con 0x7fa4541321a0 2024-08-16T02:37:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7f8bb5209640 1 --2- 172.21.15.116:0/1032090236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb0131c10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=105e4edbfd9db209 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa45c464640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa45409eab0 con 0x7fa454132cc0 2024-08-16T02:37:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa45a200640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa4541321a0 0x7fa4541325b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45004/0 (socket says 172.21.15.116:45004) 2024-08-16T02:37:12.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa45a200640 1 -- 172.21.15.116:0/3734568611 learned_addr learned my addr 172.21.15.116:0/3734568611 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 902708582 0 0) 0x7fc8ec09e750 con 0x7fc8ec09bf60 2024-08-16T02:37:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8d4003580 con 0x7fc8ec09bf60 2024-08-16T02:37:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450362091 0 0) 0x7fa45409dc90 con 0x7fa4541321a0 2024-08-16T02:37:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.490+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa440003580 con 0x7fa4541321a0 2024-08-16T02:37:12.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57559a640 1 -- 172.21.15.116:0/277747350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd56c00e4d0 con 0x7fd5781384c0 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4104355532 0 0) 0x7fc8d4003580 con 0x7fc8ec09bf60 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 msgr2=0x7fc8ec09ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 --2- 172.21.15.116:0/1466599527 >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 0x7fc8ec09ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec074770 con 0x7fc8ec09bf60 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 1177384933 0 0) 0x7fc8e0001620 con 0x7fc8ec09bf60 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 >> v1:172.21.15.116:6789/0 conn(0x7fc8ec09bf60 legacy=0x7fc8ec132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 --2- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8d4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec09e9f0 con 0x7fc8d4003a70 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1310231687 0 0) 0x7fa440003580 con 0x7fa4541321a0 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 >> v2:172.21.15.116:3300/0 conn(0x7fa454132cc0 msgr2=0x7fa4541376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 --2- 172.21.15.116:0/3734568611 >> v2:172.21.15.116:3300/0 conn(0x7fa454132cc0 0x7fa4541376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45406d990 con 0x7fa4541321a0 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 205+0+0 (unknown 1177384933 0 0) 0x7fa450001620 con 0x7fa4541321a0 2024-08-16T02:37:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fc8eaffd640 1 --2- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8d4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 >> v1:172.21.15.116:6789/0 conn(0x7fa4541321a0 legacy=0x7fa4541325b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 --2- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa440003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa45409eab0 con 0x7fa440003a20 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fa45a200640 1 --2- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa440003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 -- 172.21.15.116:0/277747350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5781384c0 msgr2=0x7fd57813a8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/277747350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5781384c0 0x7fd57813a8e0 crc :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7f8ba7fff640 1 -- 172.21.15.116:0/1032090236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8bac009d90 con 0x7f8bb009fb90 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 -- 172.21.15.116:0/277747350 shutdown_connections 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/277747350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd5781384c0 0x7fd57813a8e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 -- 172.21.15.116:0/277747350 >> 172.21.15.116:0/277747350 conn(0x7fd5780a5960 msgr2=0x7fd5780a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 -- 172.21.15.116:0/277747350 shutdown_connections 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.491+0000 7fd57cc6d640 1 -- 172.21.15.116:0/277747350 wait complete. 2024-08-16T02:37:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fc8eaffd640 1 -- 172.21.15.116:0/1466599527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d4003580 con 0x7fc8d4003a70 2024-08-16T02:37:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7f8ba7fff640 1 -- 172.21.15.116:0/1032090236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8bac00e230 con 0x7f8bb009fb90 2024-08-16T02:37:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7f8ba7fff640 1 -- 172.21.15.116:0/1032090236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8bac00e4d0 con 0x7f8bb009fb90 2024-08-16T02:37:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fa45a200640 1 -- 172.21.15.116:0/3734568611 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45406d990 con 0x7fa440003a20 2024-08-16T02:37:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fd57cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fc8eaffd640 1 --2- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8d4003ec0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1af7f1c0afd58 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fa45a200640 1 --2- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa440003e70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5fa1f6af86a384 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa450007d70 con 0x7fa440003a20 2024-08-16T02:37:12.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc8dc0097b0 con 0x7fc8d4003a70 2024-08-16T02:37:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa450005930 con 0x7fa440003a20 2024-08-16T02:37:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fa4591fe640 1 -- 172.21.15.116:0/3734568611 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa450005bd0 con 0x7fa440003a20 2024-08-16T02:37:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc8dc009e50 con 0x7fc8d4003a70 2024-08-16T02:37:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.492+0000 7fc8e9ffb640 1 -- 172.21.15.116:0/1466599527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc8dc0097b0 con 0x7fc8d4003a70 2024-08-16T02:37:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 -- 172.21.15.116:0/1032090236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 msgr2=0x7f8bb0131c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 --2- 172.21.15.116:0/1032090236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb0131c10 crc :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 -- 172.21.15.116:0/1032090236 shutdown_connections 2024-08-16T02:37:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 --2- 172.21.15.116:0/1032090236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb0131c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 -- 172.21.15.116:0/1032090236 >> 172.21.15.116:0/1032090236 conn(0x7f8bb00a1550 msgr2=0x7f8bb00a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 -- 172.21.15.116:0/1032090236 shutdown_connections 2024-08-16T02:37:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 1 -- 172.21.15.116:0/1032090236 wait complete. 2024-08-16T02:37:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 -- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 msgr2=0x7fa440003e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 --2- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa440003e70 crc :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 msgr2=0x7fc8d4003ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 --2- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8d4003ec0 crc :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 -- 172.21.15.116:0/3734568611 shutdown_connections 2024-08-16T02:37:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 --2- 172.21.15.116:0/3734568611 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa440003e70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 --2- 172.21.15.116:0/3734568611 >> v2:172.21.15.116:3300/0 conn(0x7fa454132cc0 0x7fa4541376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 -- 172.21.15.116:0/3734568611 >> 172.21.15.116:0/3734568611 conn(0x7fa45409f1d0 msgr2=0x7fa45409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 shutdown_connections 2024-08-16T02:37:12.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7f8bb746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 --2- 172.21.15.116:0/1466599527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8d4003ec0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 --2- 172.21.15.116:0/1466599527 >> v2:172.21.15.116:3300/0 conn(0x7fc8ec09d5f0 0x7fc8ec09ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 -- 172.21.15.116:0/3734568611 shutdown_connections 2024-08-16T02:37:12.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fa45c464640 1 -- 172.21.15.116:0/3734568611 wait complete. 2024-08-16T02:37:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.493+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 >> 172.21.15.116:0/1466599527 conn(0x7fc8ec06caf0 msgr2=0x7fc8ec06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 shutdown_connections 2024-08-16T02:37:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fd57cc6d640 1 Processor -- start 2024-08-16T02:37:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fc8f1464640 1 -- 172.21.15.116:0/1466599527 wait complete. 2024-08-16T02:37:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fd57cc6d640 1 -- start start 2024-08-16T02:37:12.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fa45c464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.494+0000 7fc8f1464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57cc6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57cc6d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd578122e60 con 0x7fd578135570 2024-08-16T02:37:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47598/0 (socket says 172.21.15.116:47598) 2024-08-16T02:37:12.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57659c640 1 -- 172.21.15.116:0/2330274803 learned_addr learned my addr 172.21.15.116:0/2330274803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fa45c464640 1 Processor -- start 2024-08-16T02:37:12.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fa45c464640 1 -- start start 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57659c640 1 -- 172.21.15.116:0/2330274803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd578134510 con 0x7fd578135570 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.495+0000 7fd57659c640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd568005a60 con 0x7fd578135570 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd568005bc0 con 0x7fd578135570 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd578120190 con 0x7fd578135570 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd568005e60 con 0x7fd578135570 2024-08-16T02:37:12.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd578121690 con 0x7fd578135570 2024-08-16T02:37:12.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7f8bb746d640 1 Processor -- start 2024-08-16T02:37:12.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.496+0000 7f8bb746d640 1 -- start start 2024-08-16T02:37:12.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fc8f1464640 1 Processor -- start 2024-08-16T02:37:12.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fc8f1464640 1 -- start start 2024-08-16T02:37:12.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45c464640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45a200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47614/0 (socket says 172.21.15.116:47614) 2024-08-16T02:37:12.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5780a3f90 con 0x7fd578135570 2024-08-16T02:37:12.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45a200640 1 -- 172.21.15.116:0/4285600414 learned_addr learned my addr 172.21.15.116:0/4285600414 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa454120520 con 0x7fa440003a20 2024-08-16T02:37:12.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fd5680070b0 con 0x7fd578135570 2024-08-16T02:37:12.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.497+0000 7fa45a200640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45409dec0 con 0x7fa440003a20 2024-08-16T02:37:12.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.498+0000 7fa45a200640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.498+0000 7fd566ffd640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 0x7fd558040200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.498+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fd56804e610 con 0x7fd578135570 2024-08-16T02:37:12.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.498+0000 7fd575d9b640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 0x7fd558040200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fd575d9b640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 0x7fd558040200 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7f8bb746d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8f1464640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7f8bb746d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8bb011cad0 con 0x7f8bb009fb90 2024-08-16T02:37:12.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8f1464640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ec122230 con 0x7fc8d4003a70 2024-08-16T02:37:12.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7f8bb5209640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7f8bb5209640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47626/0 (socket says 172.21.15.116:47626) 2024-08-16T02:37:12.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7f8bb5209640 1 -- 172.21.15.116:0/3760404212 learned_addr learned my addr 172.21.15.116:0/3760404212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8eaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47628/0 (socket says 172.21.15.116:47628) 2024-08-16T02:37:12.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8eaffd640 1 -- 172.21.15.116:0/3476148297 learned_addr learned my addr 172.21.15.116:0/3476148297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.499+0000 7fc8eaffd640 1 -- 172.21.15.116:0/3476148297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ec11fcd0 con 0x7fc8d4003a70 2024-08-16T02:37:12.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.500+0000 7fc8eaffd640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.500+0000 7f8bb5209640 1 -- 172.21.15.116:0/3760404212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bb011e5a0 con 0x7f8bb009fb90 2024-08-16T02:37:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.500+0000 7f8bb5209640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8bac013400 con 0x7f8bb009fb90 2024-08-16T02:37:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc8dc0097b0 con 0x7fc8d4003a70 2024-08-16T02:37:12.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fd56800d930 con 0x7fd578135570 2024-08-16T02:37:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa450004970 con 0x7fa440003a20 2024-08-16T02:37:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bb011c0b0 con 0x7f8bb009fb90 2024-08-16T02:37:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fa450004ad0 con 0x7fa440003a20 2024-08-16T02:37:12.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fc8f1464640 1 -- 172.21.15.116:0/3476148297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ec123850 con 0x7fc8d4003a70 2024-08-16T02:37:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45411cb30 con 0x7fa440003a20 2024-08-16T02:37:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc8dc003c00 con 0x7fc8d4003a70 2024-08-16T02:37:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc8dc003ea0 con 0x7fc8d4003a70 2024-08-16T02:37:12.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fa450004d70 con 0x7fa440003a20 2024-08-16T02:37:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fc8f1464640 1 -- 172.21.15.116:0/3476148297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ec122ce0 con 0x7fc8d4003a70 2024-08-16T02:37:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bb012bdc0 con 0x7f8bb009fb90 2024-08-16T02:37:12.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.503+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45411f910 con 0x7fa440003a20 2024-08-16T02:37:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.504+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f8bac0093c0 con 0x7f8bb009fb90 2024-08-16T02:37:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.505+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f8bac013400 con 0x7f8bb009fb90 2024-08-16T02:37:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.505+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8c40050f0 con 0x7fc8d4003a70 2024-08-16T02:37:12.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.505+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b900050f0 con 0x7f8bb009fb90 2024-08-16T02:37:12.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fc8dc015420 con 0x7fc8d4003a70 2024-08-16T02:37:12.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.505+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fa450005250 con 0x7fa440003a20 2024-08-16T02:37:12.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fa44affd640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 0x7fa4380401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fa45004fff0 con 0x7fa440003a20 2024-08-16T02:37:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fc8d37fe640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 0x7fc8d8042420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fc8dc04fc70 con 0x7fc8d4003a70 2024-08-16T02:37:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa45409dc90 con 0x7fa440003a20 2024-08-16T02:37:12.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fc8ea7fc640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 0x7fc8d8042420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7fa4599ff640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 0x7fa4380401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f8bac013560 con 0x7f8bb009fb90 2024-08-16T02:37:12.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7f8ba5ffb640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 0x7f8b94040210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.506+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f8bac018040 con 0x7f8bb009fb90 2024-08-16T02:37:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.507+0000 7f8bb4a08640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 0x7f8b94040210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.507+0000 7fa4599ff640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 0x7fa4380401d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.507+0000 7fc8ea7fc640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 0x7fc8d8042420 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.507+0000 7f8bb4a08640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 0x7f8b94040210 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.517+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f8bac0123f0 con 0x7f8bb009fb90 2024-08-16T02:37:12.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.527+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fc8dc00f600 con 0x7fc8d4003a70 2024-08-16T02:37:12.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.538+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fa4500153b0 con 0x7fa440003a20 2024-08-16T02:37:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.709+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8b90005670 con 0x7f8bb009fb90 2024-08-16T02:37:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.710+0000 7f8ba5ffb640 1 -- 172.21.15.116:0/3760404212 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7f8bac000ed0 con 0x7f8bb009fb90 2024-08-16T02:37:12.712 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 msgr2=0x7f8b94040210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 0x7f8b94040210 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 msgr2=0x7f8bb01c5910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 crc :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 shutdown_connections 2024-08-16T02:37:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f8b9403dd00 0x7f8b94040210 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 --2- 172.21.15.116:0/3760404212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bb009fb90 0x7f8bb01c5910 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 >> 172.21.15.116:0/3760404212 conn(0x7f8bb00a1550 msgr2=0x7f8bb012f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 shutdown_connections 2024-08-16T02:37:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.712+0000 7f8bb746d640 1 -- 172.21.15.116:0/3760404212 wait complete. 2024-08-16T02:37:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.713+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fd578134510 con 0x7fd578135570 2024-08-16T02:37:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.713+0000 7fd566ffd640 1 -- 172.21.15.116:0/2330274803 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fd578134510 con 0x7fd578135570 2024-08-16T02:37:12.716 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 msgr2=0x7fd558040200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 0x7fd558040200 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 msgr2=0x7fd578131d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 crc :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 shutdown_connections 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd55803dcf0 0x7fd558040200 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 --2- 172.21.15.116:0/2330274803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd578135570 0x7fd578131d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 >> 172.21.15.116:0/2330274803 conn(0x7fd5780a5960 msgr2=0x7fd5781362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 shutdown_connections 2024-08-16T02:37:12.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.717+0000 7fd57cc6d640 1 -- 172.21.15.116:0/2330274803 wait complete. 2024-08-16T02:37:12.729 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.0 2024-08-16T02:37:12.730 DEBUG:teuthology.parallel:result is None 2024-08-16T02:37:12.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.729+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc8c40053a0 con 0x7fc8d4003a70 2024-08-16T02:37:12.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.729+0000 7fc8d37fe640 1 -- 172.21.15.116:0/3476148297 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fc8dc00f240 con 0x7fc8d4003a70 2024-08-16T02:37:12.731 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 msgr2=0x7fc8d8042420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 0x7fc8d8042420 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 msgr2=0x7fc8ec120070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 crc :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 shutdown_connections 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc8d803ff10 0x7fc8d8042420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 --2- 172.21.15.116:0/3476148297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4003a70 0x7fc8ec120070 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.731+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 >> 172.21.15.116:0/3476148297 conn(0x7fc8ec06caf0 msgr2=0x7fc8ec071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.732+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 shutdown_connections 2024-08-16T02:37:12.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.732+0000 7fc8d17fa640 1 -- 172.21.15.116:0/3476148297 wait complete. 2024-08-16T02:37:12.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.3 2024-08-16T02:37:12.735 DEBUG:teuthology.parallel:result is None 2024-08-16T02:37:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.744+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa45411ccc0 con 0x7fa440003a20 2024-08-16T02:37:12.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.2 2024-08-16T02:37:12.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.744+0000 7fa44affd640 1 -- 172.21.15.116:0/4285600414 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (crc 0 0 0) 0x7fa4500122c0 con 0x7fa440003a20 2024-08-16T02:37:12.746 DEBUG:teuthology.parallel:result is None 2024-08-16T02:37:12.747 INFO:teuthology.orchestra.run.smithi116.stdout:38654705672 2024-08-16T02:37:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 msgr2=0x7fa4380401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 0x7fa4380401d0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 msgr2=0x7fa45411ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 crc :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 shutdown_connections 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fa43803dcc0 0x7fa4380401d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 --2- 172.21.15.116:0/4285600414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa440003a20 0x7fa45411ced0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 >> 172.21.15.116:0/4285600414 conn(0x7fa45409f1d0 msgr2=0x7fa45409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 shutdown_connections 2024-08-16T02:37:12.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.746+0000 7fa45c464640 1 -- 172.21.15.116:0/4285600414 wait complete. 2024-08-16T02:37:12.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.1 2024-08-16T02:37:12.759 DEBUG:teuthology.parallel:result is None 2024-08-16T02:37:12.759 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:37:12.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:37:12.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.903+0000 7fe0ef352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.907+0000 7fe0ef352640 1 Processor -- start 2024-08-16T02:37:12.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.907+0000 7fe0ef352640 1 -- start start 2024-08-16T02:37:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ef352640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0e81363a0 0x7fe0e81387c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ef352640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e8133f80 con 0x7fe0e8133460 2024-08-16T02:37:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ef352640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e80a3080 con 0x7fe0e81363a0 2024-08-16T02:37:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0e81363a0 0x7fe0e81387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0e81363a0 0x7fe0e81387c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47658/0 (socket says 172.21.15.116:47658) 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 -- 172.21.15.116:0/703516287 learned_addr learned my addr 172.21.15.116:0/703516287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ed0ee640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe0e8133460 0x7fe0e8133870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45014/0 (socket says 172.21.15.116:45014) 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 -- 172.21.15.116:0/703516287 >> v1:172.21.15.116:6789/0 conn(0x7fe0e8133460 legacy=0x7fe0e8133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 -- 172.21.15.116:0/703516287 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e8133030 con 0x7fe0e81363a0 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0ec8ed640 1 --2- 172.21.15.116:0/703516287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e81363a0 0x7fe0e81387c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a3dace5857013f server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0e7fff640 1 -- 172.21.15.116:0/703516287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe0dc009e50 con 0x7fe0e81363a0 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0e7fff640 1 -- 172.21.15.116:0/703516287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe0dc00e230 con 0x7fe0e81363a0 2024-08-16T02:37:12.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.909+0000 7fe0e7fff640 1 -- 172.21.15.116:0/703516287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe0dc00e4d0 con 0x7fe0e81363a0 2024-08-16T02:37:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 -- 172.21.15.116:0/703516287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e81363a0 msgr2=0x7fe0e81387c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 --2- 172.21.15.116:0/703516287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e81363a0 0x7fe0e81387c0 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 -- 172.21.15.116:0/703516287 shutdown_connections 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 --2- 172.21.15.116:0/703516287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e81363a0 0x7fe0e81387c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 -- 172.21.15.116:0/703516287 >> 172.21.15.116:0/703516287 conn(0x7fe0e80a1550 msgr2=0x7fe0e80a1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 -- 172.21.15.116:0/703516287 shutdown_connections 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 1 -- 172.21.15.116:0/703516287 wait complete. 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.910+0000 7fe0ef352640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.912+0000 7fe0ef352640 1 Processor -- start 2024-08-16T02:37:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.912+0000 7fe0ef352640 1 -- start start 2024-08-16T02:37:12.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ef352640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ef352640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e8130ca0 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ed0ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ed0ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47666/0 (socket says 172.21.15.116:47666) 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ed0ee640 1 -- 172.21.15.116:0/947625482 learned_addr learned my addr 172.21.15.116:0/947625482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ed0ee640 1 -- 172.21.15.116:0/947625482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e811ed40 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ed0ee640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe0d80059a0 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe0d8005b00 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e811f320 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.913+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe0d8005da0 con 0x7fe0e8133460 2024-08-16T02:37:12.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.914+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e811fbd0 con 0x7fe0e8133460 2024-08-16T02:37:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.915+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fe0d80070b0 con 0x7fe0e8133460 2024-08-16T02:37:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.915+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e809f7d0 con 0x7fe0e8133460 2024-08-16T02:37:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.915+0000 7fe0e5ffb640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 0x7fe0d4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.915+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fe0d804e740 con 0x7fe0e8133460 2024-08-16T02:37:12.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.915+0000 7fe0ec8ed640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 0x7fe0d4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:12.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.916+0000 7fe0ec8ed640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 0x7fe0d4040220 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:12.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:12.920+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fe0d800de40 con 0x7fe0e8133460 2024-08-16T02:37:13.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.105+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe0e811ebd0 con 0x7fe0d403dd10 2024-08-16T02:37:13.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.106+0000 7fe0e5ffb640 1 -- 172.21.15.116:0/947625482 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7fe0e811ebd0 con 0x7fe0d403dd10 2024-08-16T02:37:13.109 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:13.109 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:37:13.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 msgr2=0x7fe0d4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 0x7fe0d4040220 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 msgr2=0x7fe0e811d8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 crc :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 shutdown_connections 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe0d403dd10 0x7fe0d4040220 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 --2- 172.21.15.116:0/947625482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0e8133460 0x7fe0e811d8a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 >> 172.21.15.116:0/947625482 conn(0x7fe0e80a1550 msgr2=0x7fe0e809ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 shutdown_connections 2024-08-16T02:37:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.108+0000 7fe0ef352640 1 -- 172.21.15.116:0/947625482 wait complete. 2024-08-16T02:37:13.121 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-08-16T02:37:12.290246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777558+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:37:10.777558+0000","last_peered":"2024-08-16T02:37:10.777558+0000","last_clean":"2024-08-16T02:37:10.777558+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:37:10.777558+0000","last_undegraded":"2024-08-16T02:37:10.777558+0000","last_fullsized":"2024-08-16T02:37:10.777558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777710+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:37:10.777710+0000","last_peered":"2024-08-16T02:37:10.777710+0000","last_clean":"2024-08-16T02:37:10.777710+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:37:10.777710+0000","last_undegraded":"2024-08-16T02:37:10.777710+0000","last_fullsized":"2024-08-16T02:37:10.777710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.774047+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:37:10.774047+0000","last_peered":"2024-08-16T02:37:10.774047+0000","last_clean":"2024-08-16T02:37:10.774047+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:37:10.774047+0000","last_undegraded":"2024-08-16T02:37:10.774047+0000","last_fullsized":"2024-08-16T02:37:10.774047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771968+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:37:10.771968+0000","last_peered":"2024-08-16T02:37:10.771968+0000","last_clean":"2024-08-16T02:37:10.771968+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:37:10.771968+0000","last_undegraded":"2024-08-16T02:37:10.771968+0000","last_fullsized":"2024-08-16T02:37:10.771968+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.776780+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:37:10.776780+0000","last_peered":"2024-08-16T02:37:10.776780+0000","last_clean":"2024-08-16T02:37:10.776780+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:37:10.776780+0000","last_undegraded":"2024-08-16T02:37:10.776780+0000","last_fullsized":"2024-08-16T02:37:10.776780+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771915+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:37:10.771915+0000","last_peered":"2024-08-16T02:37:10.771915+0000","last_clean":"2024-08-16T02:37:10.771915+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:37:10.771915+0000","last_undegraded":"2024-08-16T02:37:10.771915+0000","last_fullsized":"2024-08-16T02:37:10.771915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.773680+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:37:10.773680+0000","last_peered":"2024-08-16T02:37:10.773680+0000","last_clean":"2024-08-16T02:37:10.773680+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:37:10.773680+0000","last_undegraded":"2024-08-16T02:37:10.773680+0000","last_fullsized":"2024-08-16T02:37:10.773680+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777297+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:37:10.777297+0000","last_peered":"2024-08-16T02:37:10.777297+0000","last_clean":"2024-08-16T02:37:10.777297+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:37:10.777297+0000","last_undegraded":"2024-08-16T02:37:10.777297+0000","last_fullsized":"2024-08-16T02:37:10.777297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771368+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:37:10.771368+0000","last_peered":"2024-08-16T02:37:10.771368+0000","last_clean":"2024-08-16T02:37:10.771368+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:37:10.771368+0000","last_undegraded":"2024-08-16T02:37:10.771368+0000","last_fullsized":"2024-08-16T02:37:10.771368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:37:13.122 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:37:13.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.225+0000 7fae2c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.230+0000 7fae2c26d640 1 Processor -- start 2024-08-16T02:37:13.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.230+0000 7fae2c26d640 1 -- start start 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.231+0000 7fae2c26d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fae2409e340 0x7fae24138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.231+0000 7fae2c26d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae240a0f70 con 0x7fae2406f660 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.231+0000 7fae2c26d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fae240a0d00 con 0x7fae2409e340 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae2a009640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fae2406f660 0x7fae2409dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45018/0 (socket says 172.21.15.116:45018) 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae2a009640 1 -- 172.21.15.116:0/3807919331 learned_addr learned my addr 172.21.15.116:0/3807919331 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29808640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fae2409e340 0x7fae24138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29808640 1 -- 172.21.15.116:0/3807919331 >> v1:172.21.15.116:6789/0 conn(0x7fae2406f660 legacy=0x7fae2409dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29808640 1 -- 172.21.15.116:0/3807919331 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2409ee50 con 0x7fae2409e340 2024-08-16T02:37:13.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29808640 1 --2- 172.21.15.116:0/3807919331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2409e340 0x7fae24138540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d5f19f5e7fe47d server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29007640 1 -- 172.21.15.116:0/3807919331 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fae14009d90 con 0x7fae2409e340 2024-08-16T02:37:13.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29007640 1 -- 172.21.15.116:0/3807919331 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fae1400e230 con 0x7fae2409e340 2024-08-16T02:37:13.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.232+0000 7fae29007640 1 -- 172.21.15.116:0/3807919331 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fae1400e4d0 con 0x7fae2409e340 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 -- 172.21.15.116:0/3807919331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2409e340 msgr2=0x7fae24138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 --2- 172.21.15.116:0/3807919331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2409e340 0x7fae24138540 crc :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 -- 172.21.15.116:0/3807919331 shutdown_connections 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 --2- 172.21.15.116:0/3807919331 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2409e340 0x7fae24138540 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 -- 172.21.15.116:0/3807919331 >> 172.21.15.116:0/3807919331 conn(0x7fae2409f1d0 msgr2=0x7fae2409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:13.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 -- 172.21.15.116:0/3807919331 shutdown_connections 2024-08-16T02:37:13.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 1 -- 172.21.15.116:0/3807919331 wait complete. 2024-08-16T02:37:13.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.233+0000 7fae2c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:13.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.234+0000 7fae2c26d640 1 Processor -- start 2024-08-16T02:37:13.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.235+0000 7fae2c26d640 1 -- start start 2024-08-16T02:37:13.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.235+0000 7fae2c26d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.235+0000 7fae2c26d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fae241251f0 con 0x7fae2406f660 2024-08-16T02:37:13.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2a009640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2a009640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47678/0 (socket says 172.21.15.116:47678) 2024-08-16T02:37:13.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2a009640 1 -- 172.21.15.116:0/1511212430 learned_addr learned my addr 172.21.15.116:0/1511212430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2a009640 1 -- 172.21.15.116:0/1511212430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2409ba20 con 0x7fae2406f660 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2a009640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fae20005a60 con 0x7fae2406f660 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fae20005bc0 con 0x7fae2406f660 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae241269a0 con 0x7fae2406f660 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fae20005e60 con 0x7fae2406f660 2024-08-16T02:37:13.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.236+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae241265e0 con 0x7fae2406f660 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.237+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae2409dc30 con 0x7fae2406f660 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.237+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fae2000c7f0 con 0x7fae2406f660 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.237+0000 7fae1affd640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 0x7fae08040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.238+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7fae2004e740 con 0x7fae2406f660 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.238+0000 7fae29808640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 0x7fae08040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.238+0000 7fae29808640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 0x7fae08040220 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.243+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fae2000d7f0 con 0x7fae2406f660 2024-08-16T02:37:13.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.424+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae2409bdf0 con 0x7fae0803dd10 2024-08-16T02:37:13.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.426+0000 7fae1affd640 1 -- 172.21.15.116:0/1511212430 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31201 (crc 0 0 0) 0x7fae2409bdf0 con 0x7fae0803dd10 2024-08-16T02:37:13.428 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:13.428 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 msgr2=0x7fae08040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 0x7fae08040220 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 msgr2=0x7fae241c99c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 crc :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 shutdown_connections 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fae0803dd10 0x7fae08040220 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 --2- 172.21.15.116:0/1511212430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fae2406f660 0x7fae241c99c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 >> 172.21.15.116:0/1511212430 conn(0x7fae2409f1d0 msgr2=0x7fae240a2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:13.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 shutdown_connections 2024-08-16T02:37:13.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.428+0000 7fae2c26d640 1 -- 172.21.15.116:0/1511212430 wait complete. 2024-08-16T02:37:13.440 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-08-16T02:37:12.290246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":3664,"kb_used_data":3664,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209187248,"statfs":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777558+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:37:10.777558+0000","last_peered":"2024-08-16T02:37:10.777558+0000","last_clean":"2024-08-16T02:37:10.777558+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:37:10.777558+0000","last_undegraded":"2024-08-16T02:37:10.777558+0000","last_fullsized":"2024-08-16T02:37:10.777558+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777710+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:37:10.777710+0000","last_peered":"2024-08-16T02:37:10.777710+0000","last_clean":"2024-08-16T02:37:10.777710+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:37:10.777710+0000","last_undegraded":"2024-08-16T02:37:10.777710+0000","last_fullsized":"2024-08-16T02:37:10.777710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.774047+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:37:10.774047+0000","last_peered":"2024-08-16T02:37:10.774047+0000","last_clean":"2024-08-16T02:37:10.774047+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:37:10.774047+0000","last_undegraded":"2024-08-16T02:37:10.774047+0000","last_fullsized":"2024-08-16T02:37:10.774047+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771968+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:37:10.771968+0000","last_peered":"2024-08-16T02:37:10.771968+0000","last_clean":"2024-08-16T02:37:10.771968+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:37:10.771968+0000","last_undegraded":"2024-08-16T02:37:10.771968+0000","last_fullsized":"2024-08-16T02:37:10.771968+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.776780+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:37:10.776780+0000","last_peered":"2024-08-16T02:37:10.776780+0000","last_clean":"2024-08-16T02:37:10.776780+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:37:10.776780+0000","last_undegraded":"2024-08-16T02:37:10.776780+0000","last_fullsized":"2024-08-16T02:37:10.776780+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771915+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:37:10.771915+0000","last_peered":"2024-08-16T02:37:10.771915+0000","last_clean":"2024-08-16T02:37:10.771915+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:37:10.771915+0000","last_undegraded":"2024-08-16T02:37:10.771915+0000","last_fullsized":"2024-08-16T02:37:10.771915+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.773680+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:37:10.773680+0000","last_peered":"2024-08-16T02:37:10.773680+0000","last_clean":"2024-08-16T02:37:10.773680+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:37:10.773680+0000","last_undegraded":"2024-08-16T02:37:10.773680+0000","last_fullsized":"2024-08-16T02:37:10.773680+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.777297+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:37:10.777297+0000","last_peered":"2024-08-16T02:37:10.777297+0000","last_clean":"2024-08-16T02:37:10.777297+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:37:10.777297+0000","last_undegraded":"2024-08-16T02:37:10.777297+0000","last_fullsized":"2024-08-16T02:37:10.777297+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-08-16T02:37:10.771368+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:37:10.771368+0000","last_peered":"2024-08-16T02:37:10.771368+0000","last_clean":"2024-08-16T02:37:10.771368+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:37:10.771368+0000","last_undegraded":"2024-08-16T02:37:10.771368+0000","last_fullsized":"2024-08-16T02:37:10.771368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214207741952,"internally_reserved":0,"allocated":3751936,"data_stored":3751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1172,"kb_used_data":1172,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296556,"statfs":{"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":532,"kb_used_data":532,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297196,"statfs":{"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":528,"kb_used_data":528,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297200,"statfs":{"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1432,"kb_used_data":1432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296296,"statfs":{"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551407104,"internally_reserved":0,"allocated":1466368,"data_stored":1466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53552332800,"internally_reserved":0,"allocated":540672,"data_stored":540672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552328704,"internally_reserved":0,"allocated":544768,"data_stored":544768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53551673344,"internally_reserved":0,"allocated":1200128,"data_stored":1200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:37:13.441 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:37:13.441 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-16T02:37:13.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-16T02:37:13.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-16T02:37:13.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.544+0000 7f4360a20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.547+0000 7f4360a20640 1 Processor -- start 2024-08-16T02:37:13.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.547+0000 7f4360a20640 1 -- start start 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.548+0000 7f4360a20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f435c09c540 0x7f435c13a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.548+0000 7f4360a20640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435c06f900 con 0x7f435c09ba20 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.548+0000 7f4360a20640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f435c09fff0 con 0x7f435c09c540 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.548+0000 7f4351d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f435c09c540 0x7f435c13a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.548+0000 7f435a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f435c09ba20 0x7f435c09be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45034/0 (socket says 172.21.15.116:45034) 2024-08-16T02:37:13.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f435a59c640 1 -- 172.21.15.116:0/1619382288 learned_addr learned my addr 172.21.15.116:0/1619382288 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4351d9b640 1 --2- 172.21.15.116:0/1619382288 >> v2:172.21.15.116:3300/0 conn(0x7f435c09c540 0x7f435c13a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47690/0 (socket says 172.21.15.116:47690) 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4351d9b640 1 -- 172.21.15.116:0/1619382288 >> v1:172.21.15.116:6789/0 conn(0x7f435c09ba20 legacy=0x7f435c09be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4359d9b640 1 -- 172.21.15.116:0/1619382288 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056686392 0 0) 0x7f435c06f900 con 0x7f435c09ba20 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4351d9b640 1 -- 172.21.15.116:0/1619382288 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c09d640 con 0x7f435c09c540 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4351d9b640 1 --2- 172.21.15.116:0/1619382288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09c540 0x7f435c13a720 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d22b5138eda25c server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4359d9b640 1 -- 172.21.15.116:0/1619382288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f4344009d90 con 0x7f435c09c540 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4359d9b640 1 -- 172.21.15.116:0/1619382288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f434400e230 con 0x7f435c09c540 2024-08-16T02:37:13.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.549+0000 7f4359d9b640 1 -- 172.21.15.116:0/1619382288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f434400e4d0 con 0x7f435c09c540 2024-08-16T02:37:13.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 -- 172.21.15.116:0/1619382288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09c540 msgr2=0x7f435c13a720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 --2- 172.21.15.116:0/1619382288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09c540 0x7f435c13a720 crc :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 -- 172.21.15.116:0/1619382288 shutdown_connections 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 --2- 172.21.15.116:0/1619382288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09c540 0x7f435c13a720 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 -- 172.21.15.116:0/1619382288 >> 172.21.15.116:0/1619382288 conn(0x7f435c0a09c0 msgr2=0x7f435c0a0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 -- 172.21.15.116:0/1619382288 shutdown_connections 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 1 -- 172.21.15.116:0/1619382288 wait complete. 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.550+0000 7f4360a20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.552+0000 7f4360a20640 1 Processor -- start 2024-08-16T02:37:13.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.552+0000 7f4360a20640 1 -- start start 2024-08-16T02:37:13.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f4360a20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f4360a20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f435c1ac600 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f435a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f435a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47694/0 (socket says 172.21.15.116:47694) 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f435a59c640 1 -- 172.21.15.116:0/1986673356 learned_addr learned my addr 172.21.15.116:0/1986673356 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f435a59c640 1 -- 172.21.15.116:0/1986673356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c1a9e70 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.553+0000 7f435a59c640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.554+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f434c004b80 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.554+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f434c004ce0 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.554+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c1aa180 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.554+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f434c007120 con 0x7f435c09ba20 2024-08-16T02:37:13.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.554+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1ad140 con 0x7f435c09ba20 2024-08-16T02:37:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.555+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c1ac450 con 0x7f435c09ba20 2024-08-16T02:37:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.555+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f434c0072d0 con 0x7f435c09ba20 2024-08-16T02:37:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.555+0000 7f43537fe640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 0x7f433c040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:37:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.555+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 2728+0+0 (crc 0 0 0) 0x7f434c04e870 con 0x7f435c09ba20 2024-08-16T02:37:13.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.555+0000 7f4351d9b640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 0x7f433c040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:37:13.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.556+0000 7f4351d9b640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 0x7f433c040250 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:37:13.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.560+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f434c00adb0 con 0x7f435c09ba20 2024-08-16T02:37:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.788+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f435c1ab230 con 0x7f435c09ba20 2024-08-16T02:37:13.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.789+0000 7f43537fe640 1 -- 172.21.15.116:0/1986673356 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (crc 0 0 0) 0x7f434c00d150 con 0x7f435c09ba20 2024-08-16T02:37:13.791 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:37:13.791 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 msgr2=0x7f433c040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 0x7f433c040250 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 msgr2=0x7f435c1aa520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 crc :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 shutdown_connections 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f433c03dd40 0x7f433c040250 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 --2- 172.21.15.116:0/1986673356 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f435c09ba20 0x7f435c1aa520 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 >> 172.21.15.116:0/1986673356 conn(0x7f435c0a09c0 msgr2=0x7f435c1a3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 shutdown_connections 2024-08-16T02:37:13.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:37:13.791+0000 7f4360a20640 1 -- 172.21.15.116:0/1986673356 wait complete. 2024-08-16T02:37:13.804 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-16T02:37:13.804 INFO:teuthology.run_tasks:Running task workunit... 2024-08-16T02:37:13.817 INFO:tasks.workunit:Pulling workunits from ref 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:37:13.818 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-16T02:37:13.818 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:37:13.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:37:13.834 INFO:teuthology.orchestra.run.smithi116.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-16T02:37:13.835 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-16T02:37:13.890 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-16T02:37:13.890 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-16T02:37:13.945 INFO:tasks.workunit:timeout=3h 2024-08-16T02:37:13.946 INFO:tasks.workunit:cleanup=True 2024-08-16T02:37:13.946 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 6f386a3f0b1fe6a4d7219b54bf963d790bc84004 2024-08-16T02:37:14.001 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-16T02:38:11.726 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to '6f386a3f0b1fe6a4d7219b54bf963d790bc84004'. 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-16T02:38:11.727 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-08-16T02:38:11.728 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at 6f386a3f0b1 Merge branch 'wip-seastore-more-reports' of https://github.com/cyx1231st/ceph into wip-yingxin-testing-2024-08-15-1551 2024-08-16T02:38:11.730 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-16T02:38:11.827 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-16T02:38:11.829 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:38:11.829 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-16T02:38:12.540 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-16T02:38:12.596 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-16T02:38:12.637 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-16T02:38:12.639 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:38:12.639 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-16T02:38:12.678 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-16T02:38:12.681 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-08-16T02:38:12.682 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-16T02:38:12.697 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-16T02:38:12.698 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-16T02:38:12.698 DEBUG:teuthology.orchestra.run.smithi116:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson 2024-08-16T02:38:12.757 INFO:tasks.workunit.client.0.smithi116.stderr:+ parallel=1 2024-08-16T02:38:12.757 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' --crimson = --serial ']' 2024-08-16T02:38:12.757 INFO:tasks.workunit.client.0.smithi116.stderr:+ crimson=0 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' --crimson = --crimson ']' 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ parallel=0 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ crimson=1 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ color= 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' -t 1 ']' 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:38:12.758 INFO:tasks.workunit.client.0.smithi116.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:38:12.759 INFO:tasks.workunit.client.0.smithi116.stderr:+ declare -A pids 2024-08-16T02:38:12.759 INFO:tasks.workunit.client.0.smithi116.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-16T02:38:12.759 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' 0 -eq 1 ']' 2024-08-16T02:38:12.759 INFO:tasks.workunit.client.0.smithi116.stderr:+ ceph_test_rados_api_aio 2024-08-16T02:38:12.775 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2024-08-16T02:38:12.776 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 42 tests from 2 test suites. 2024-08-16T02:38:12.776 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2024-08-16T02:38:12.776 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 26 tests from LibRadosAio 2024-08-16T02:38:12.776 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.TooBig 2024-08-16T02:38:15.194 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.TooBig (2418 ms) 2024-08-16T02:38:15.194 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleWrite 2024-08-16T02:38:18.224 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleWrite (3030 ms) 2024-08-16T02:38:18.224 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.WaitForSafe 2024-08-16T02:38:21.229 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.WaitForSafe (3005 ms) 2024-08-16T02:38:21.230 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTrip 2024-08-16T02:38:24.250 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTrip (3021 ms) 2024-08-16T02:38:24.250 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTrip2 2024-08-16T02:38:27.264 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTrip2 (3014 ms) 2024-08-16T02:38:27.264 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTrip3 2024-08-16T02:38:30.279 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTrip3 (3014 ms) 2024-08-16T02:38:30.279 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripAppend 2024-08-16T02:38:33.299 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripAppend (3021 ms) 2024-08-16T02:38:33.300 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RemoveTest 2024-08-16T02:38:36.311 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RemoveTest (3012 ms) 2024-08-16T02:38:36.311 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-16T02:38:39.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.XattrsRoundTrip (3009 ms) 2024-08-16T02:38:39.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RmXattr 2024-08-16T02:38:42.343 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RmXattr (3023 ms) 2024-08-16T02:38:42.343 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.XattrIter 2024-08-16T02:38:45.357 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.XattrIter (3014 ms) 2024-08-16T02:38:45.357 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.IsComplete 2024-08-16T02:38:48.374 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.IsComplete (3017 ms) 2024-08-16T02:38:48.374 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.IsSafe 2024-08-16T02:38:51.390 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.IsSafe (3016 ms) 2024-08-16T02:38:51.390 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.ReturnValue 2024-08-16T02:38:54.411 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.ReturnValue (3021 ms) 2024-08-16T02:38:54.411 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.Flush 2024-08-16T02:38:57.419 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.Flush (3008 ms) 2024-08-16T02:38:57.419 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.FlushAsync 2024-08-16T02:39:00.436 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.FlushAsync (3017 ms) 2024-08-16T02:39:00.436 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-16T02:39:03.453 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteFull (3017 ms) 2024-08-16T02:39:03.454 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-16T02:39:06.466 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteSame (3013 ms) 2024-08-16T02:39:06.466 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleStat 2024-08-16T02:39:09.183 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleStat (2718 ms) 2024-08-16T02:39:09.184 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-16T02:39:12.196 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.OperateMtime (3012 ms) 2024-08-16T02:39:12.196 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.Operate2Mtime 2024-08-16T02:39:15.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.Operate2Mtime (3014 ms) 2024-08-16T02:39:15.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleStatNS 2024-08-16T02:39:18.215 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleStatNS (3005 ms) 2024-08-16T02:39:18.215 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.StatRemove 2024-08-16T02:39:21.226 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.StatRemove (3011 ms) 2024-08-16T02:39:21.226 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.ExecuteClass 2024-08-16T02:39:24.249 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.ExecuteClass (3023 ms) 2024-08-16T02:39:24.249 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.MultiWrite 2024-08-16T02:39:27.182 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.MultiWrite (2933 ms) 2024-08-16T02:39:27.182 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.AioUnlock 2024-08-16T02:39:30.203 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.AioUnlock (3018 ms) 2024-08-16T02:39:30.203 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 26 tests from LibRadosAio (77424 ms total) 2024-08-16T02:39:30.203 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:39:30.203 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 16 tests from LibRadosAioEC 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleWrite 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1100: Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite (0 ms) 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.WaitForComplete 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1132: Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete (0 ms) 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTrip 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1149: Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip (0 ms) 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTrip2 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1182: Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 (0 ms) 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1215: Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend (0 ms) 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.IsComplete 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1286: Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete (0 ms) 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.IsSafe 2024-08-16T02:39:30.205 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1326: Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe (0 ms) 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.ReturnValue 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1366: Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue (0 ms) 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.Flush 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1385: Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.Flush (0 ms) 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.FlushAsync 2024-08-16T02:39:30.206 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1415: Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync (0 ms) 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1454: Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull (0 ms) 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleStat 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1498: Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat (0 ms) 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1532: Skipped 2024-08-16T02:39:30.207 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS (0 ms) 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.StatRemove 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1593: Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove (0 ms) 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.ExecuteClass 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1649: Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass (0 ms) 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.MultiWrite 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio.cc:1681: Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite (0 ms) 2024-08-16T02:39:30.208 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 16 tests from LibRadosAioEC (0 ms total) 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 42 tests from 2 test suites ran. (77424 ms total) 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 26 tests. 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] 16 tests, listed below: 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWrite 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.WaitForComplete 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTrip2 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppend 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsComplete 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsSafe 2024-08-16T02:39:30.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValue 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.Flush 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsync 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFull 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStat 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatNS 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.StatRemove 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClass 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.MultiWrite 2024-08-16T02:39:30.210 INFO:tasks.workunit.client.0.smithi116.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-08-16T02:39:30.211 INFO:tasks.workunit.client.0.smithi116.stderr:+ '[' 0 -eq 1 ']' 2024-08-16T02:39:30.211 INFO:tasks.workunit.client.0.smithi116.stderr:+ ceph_test_rados_api_aio_pp 2024-08-16T02:39:30.223 INFO:tasks.workunit.client.0.smithi116.stdout:Running main() from gmock_main.cc 2024-08-16T02:39:30.223 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] Running 57 tests from 4 test suites. 2024-08-16T02:39:30.223 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment set-up. 2024-08-16T02:39:30.224 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 32 tests from LibRadosAio 2024-08-16T02:39:30.224 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.TooBigPP 2024-08-16T02:39:33.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.TooBigPP (2986 ms) 2024-08-16T02:39:33.209 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.PoolQuotaPP 2024-08-16T02:39:49.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.PoolQuotaPP (15994 ms) 2024-08-16T02:39:49.204 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleWritePP 2024-08-16T02:39:55.231 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleWritePP (6028 ms) 2024-08-16T02:39:55.231 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.WaitForSafePP 2024-08-16T02:39:58.240 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.WaitForSafePP (3009 ms) 2024-08-16T02:39:58.240 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripPP 2024-08-16T02:40:01.267 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripPP (3028 ms) 2024-08-16T02:40:01.268 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripPP2 2024-08-16T02:40:04.279 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripPP2 (3011 ms) 2024-08-16T02:40:04.279 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripPP3 2024-08-16T02:40:07.193 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripPP3 (2914 ms) 2024-08-16T02:40:07.193 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-16T02:40:10.212 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripSparseReadPP (3019 ms) 2024-08-16T02:40:10.212 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.IsCompletePP 2024-08-16T02:40:13.227 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.IsCompletePP (3016 ms) 2024-08-16T02:40:13.228 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.IsSafePP 2024-08-16T02:40:16.243 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.IsSafePP (3015 ms) 2024-08-16T02:40:16.243 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.ReturnValuePP 2024-08-16T02:40:19.250 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.ReturnValuePP (3007 ms) 2024-08-16T02:40:19.250 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.FlushPP 2024-08-16T02:40:22.261 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.FlushPP (3011 ms) 2024-08-16T02:40:22.262 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.FlushAsyncPP 2024-08-16T02:40:25.272 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.FlushAsyncPP (3011 ms) 2024-08-16T02:40:25.272 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-16T02:40:28.281 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP (3009 ms) 2024-08-16T02:40:28.282 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-16T02:40:31.293 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteFullPP2 (3012 ms) 2024-08-16T02:40:31.293 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-16T02:40:34.315 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP (3022 ms) 2024-08-16T02:40:34.316 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-16T02:40:37.197 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripWriteSamePP2 (2882 ms) 2024-08-16T02:40:37.197 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-16T02:40:40.205 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleStatPPNS (3008 ms) 2024-08-16T02:40:40.206 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.SimpleStatPP 2024-08-16T02:40:43.219 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.SimpleStatPP (3014 ms) 2024-08-16T02:40:43.219 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.OperateMtime 2024-08-16T02:40:46.233 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.OperateMtime (3014 ms) 2024-08-16T02:40:46.233 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.OperateMtime2 2024-08-16T02:40:49.238 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.OperateMtime2 (3005 ms) 2024-08-16T02:40:49.238 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.StatRemovePP 2024-08-16T02:40:52.258 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.StatRemovePP (3020 ms) 2024-08-16T02:40:52.259 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.ExecuteClassPP 2024-08-16T02:40:55.277 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.ExecuteClassPP (3019 ms) 2024-08-16T02:40:55.277 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.OmapPP 2024-08-16T02:40:58.286 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.OmapPP (3009 ms) 2024-08-16T02:40:58.287 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.MultiWritePP 2024-08-16T02:41:01.296 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.MultiWritePP (3010 ms) 2024-08-16T02:41:01.297 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.AioUnlockPP 2024-08-16T02:41:04.311 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.AioUnlockPP (3015 ms) 2024-08-16T02:41:04.311 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-16T02:41:07.201 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripAppendPP (2890 ms) 2024-08-16T02:41:07.201 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RacingRemovePP 2024-08-16T02:41:10.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RacingRemovePP (3015 ms) 2024-08-16T02:41:10.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-08-16T02:41:13.224 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP (3008 ms) 2024-08-16T02:41:13.224 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-08-16T02:41:16.240 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAio.RoundTripCmpExtPP2 (3016 ms) 2024-08-16T02:41:16.240 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.PoolEIOFlag 2024-08-16T02:41:18.702 INFO:tasks.workunit.client.0.smithi116.stdout:setting pool EIO 2024-08-16T02:41:28.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 689 2024-08-16T02:41:29.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 609 2024-08-16T02:41:30.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 530 2024-08-16T02:41:31.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 450 2024-08-16T02:41:32.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 370 2024-08-16T02:41:33.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 291 2024-08-16T02:41:34.364 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 210 2024-08-16T02:41:35.365 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 130 2024-08-16T02:41:36.365 INFO:tasks.workunit.client.0.smithi116.stdout:waiting for 48 2024-08-16T02:41:37.366 INFO:tasks.workunit.client.0.smithi116.stdout:max_success 1000, min_failed 0 2024-08-16T02:41:37.366 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2421: Failure 2024-08-16T02:41:37.367 INFO:tasks.workunit.client.0.smithi116.stdout:Value of: max_success + 1 == min_failed 2024-08-16T02:41:37.367 INFO:tasks.workunit.client.0.smithi116.stdout: Actual: false 2024-08-16T02:41:37.367 INFO:tasks.workunit.client.0.smithi116.stdout:Expected: true 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag (21975 ms) 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAio.MultiReads 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2427: Skipped 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAio.MultiReads (0 ms) 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 32 tests from LibRadosAio (127992 ms total) 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:41:38.215 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from LibRadosAioPP 2024-08-16T02:41:38.216 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-08-16T02:41:41.225 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAioPP.ReadIntoBufferlist (3010 ms) 2024-08-16T02:41:41.225 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-08-16T02:41:50.221 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAioPP.XattrsRoundTripPP (8996 ms) 2024-08-16T02:41:50.221 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioPP.RmXattrPP 2024-08-16T02:42:05.285 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAioPP.RmXattrPP (15064 ms) 2024-08-16T02:42:05.285 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioPP.RemoveTestPP 2024-08-16T02:42:08.297 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosAioPP.RemoveTestPP (3012 ms) 2024-08-16T02:42:08.297 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 4 tests from LibRadosAioPP (30082 ms total) 2024-08-16T02:42:08.297 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:42:08.297 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 1 test from LibRadosIoPP 2024-08-16T02:42:08.297 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosIoPP.XattrListPP 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:[ OK ] LibRadosIoPP.XattrListPP (3016 ms) 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 1 test from LibRadosIoPP (3016 ms total) 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 20 tests from LibRadosAioEC 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleWritePP 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1384: Skipped 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.318 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP (0 ms) 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.WaitForSafePP 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1419: Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP (0 ms) 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripPP 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1436: Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP (0 ms) 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripPP2 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1467: Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.319 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 (0 ms) 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripPP3 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1500: Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 (0 ms) 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1605: Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP (0 ms) 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1637: Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP (0 ms) 2024-08-16T02:42:11.320 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.IsCompletePP 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1705: Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP (0 ms) 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.IsSafePP 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1742: Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP (0 ms) 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.ReturnValuePP 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1780: Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP (0 ms) 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.FlushPP 2024-08-16T02:42:11.321 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1796: Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP (0 ms) 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.FlushAsyncPP 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1824: Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP (0 ms) 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1859: Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP (0 ms) 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-16T02:42:11.322 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1904: Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 (0 ms) 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleStatPP 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1946: Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP (0 ms) 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:1977: Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS (0 ms) 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.StatRemovePP 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2008: Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.323 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP (0 ms) 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.ExecuteClassPP 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2059: Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP (0 ms) 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.OmapPP 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2089: Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP (0 ms) 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ RUN ] LibRadosAioEC.MultiWritePP 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/test/librados/aio_cxx.cc:2124: Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:Not supported by crimson yet. Skipped 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP (0 ms) 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] 20 tests from LibRadosAioEC (0 ms total) 2024-08-16T02:42:11.324 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[----------] Global test environment tear-down 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[==========] 57 tests from 4 test suites ran. (161091 ms total) 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ PASSED ] 35 tests. 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] 21 tests, listed below: 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAio.MultiReads 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleWritePP 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.WaitForSafePP 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP2 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripPP3 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripSparseReadPP 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripAppendPP 2024-08-16T02:42:11.325 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsCompletePP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.IsSafePP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ReturnValuePP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushPP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.FlushAsyncPP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.RoundTripWriteFullPP2 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.SimpleStatPPNS 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.StatRemovePP 2024-08-16T02:42:11.326 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.ExecuteClassPP 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.OmapPP 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout:[ SKIPPED ] LibRadosAioEC.MultiWritePP 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout:[ FAILED ] 1 test, listed below: 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout:[ FAILED ] LibRadosAio.PoolEIOFlag 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stdout: 1 FAILED TEST 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl< >::~mutex_debug_impl() [with bool Recursive = false]' thread 7f8f7ea39800 time 2024-08-16T02:42:11.315098+0000 2024-08-16T02:42:11.327 INFO:tasks.workunit.client.0.smithi116.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-5812-g6f386a3f/rpm/el9/BUILD/ceph-19.0.0-5812-g6f386a3f/src/common/mutex_debug.h: 110: FAILED ceph_assert(r == 0) 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: ceph version 19.0.0-5812-g6f386a3f (6f386a3f0b1fe6a4d7219b54bf963d790bc84004) squid (dev) 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x129) [0x7f8f7dc7906e] 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7f8f7dc792b4] 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 3: ceph_test_rados_api_aio_pp() [0x4a3035] 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 4: /lib64/libc.so.6(+0x40ded) [0x7f8f7ca40ded] 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 5: on_exit() 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 6: /lib64/libc.so.6(+0x29597) [0x7f8f7ca29597] 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 7: __libc_start_main() 2024-08-16T02:42:11.328 INFO:tasks.workunit.client.0.smithi116.stderr: 8: _start() 2024-08-16T02:42:11.944 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 24: 31631 Aborted (core dumped) ceph_test_rados_$f 2024-08-16T02:42:11.945 INFO:tasks.workunit.client.0.smithi116.stderr:++ cleanup 2024-08-16T02:42:11.945 INFO:tasks.workunit.client.0.smithi116.stderr:++ pkill -P 31160 2024-08-16T02:42:11.968 INFO:tasks.workunit.client.0.smithi116.stderr:++ true 2024-08-16T02:42:11.968 INFO:tasks.workunit.client.0.smithi116.stderr:+ cleanup 2024-08-16T02:42:11.968 INFO:tasks.workunit.client.0.smithi116.stderr:+ pkill -P 31160 2024-08-16T02:42:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 134 2024-08-16T02:42:11.973 INFO:tasks.workunit.client.0.smithi116.stderr:+ true 2024-08-16T02:42:11.973 INFO:tasks.workunit:Stopping ['rados/test.sh --crimson', 'rados/test_pool_quota.sh'] on client.0... 2024-08-16T02:42:11.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-16T02:42:12.301 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi116 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-16T02:42:12.485 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b0dcd9f3e9c94a08b9c103a2da50edce Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_6f386a3f0b1fe6a4d7219b54bf963d790bc84004/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi116 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' 2024-08-16T02:42:12.487 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-16T02:42:12.496 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-16T02:42:12.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:42:12.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.605+0000 7f11eaaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:12.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.610+0000 7f11eaaad640 1 Processor -- start 2024-08-16T02:42:12.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.610+0000 7f11eaaad640 1 -- start start 2024-08-16T02:42:12.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11eaaad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e409fe30 0x7f11e40a0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11eaaad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e40a14c0 con 0x7f11e40a0780 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11eaaad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f11e406f380 con 0x7f11e409fe30 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e409fe30 0x7f11e40a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e409fe30 0x7f11e40a0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56470/0 (socket says 172.21.15.116:56470) 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11e40a0780 0x7f11e4135390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52260/0 (socket says 172.21.15.116:52260) 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 -- 172.21.15.116:0/4021458233 learned_addr learned my addr 172.21.15.116:0/4021458233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:12.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 -- 172.21.15.116:0/4021458233 >> v1:172.21.15.116:6789/0 conn(0x7f11e40a0780 legacy=0x7f11e4135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 -- 172.21.15.116:0/4021458233 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e40a2130 con 0x7f11e409fe30 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e37fe640 1 -- 172.21.15.116:0/4021458233 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3805094874 0 0) 0x7f11e40a14c0 con 0x7f11e40a0780 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.611+0000 7f11e8849640 1 --2- 172.21.15.116:0/4021458233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e40a0240 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378d6307e88c7498 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11e37fe640 1 -- 172.21.15.116:0/4021458233 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f11d4009e50 con 0x7f11e409fe30 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11e37fe640 1 -- 172.21.15.116:0/4021458233 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f11d400e230 con 0x7f11e409fe30 2024-08-16T02:42:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11e37fe640 1 -- 172.21.15.116:0/4021458233 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f11d400e4d0 con 0x7f11e409fe30 2024-08-16T02:42:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 -- 172.21.15.116:0/4021458233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 msgr2=0x7f11e40a0240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 --2- 172.21.15.116:0/4021458233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e40a0240 crc :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 -- 172.21.15.116:0/4021458233 shutdown_connections 2024-08-16T02:42:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 --2- 172.21.15.116:0/4021458233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e40a0240 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 -- 172.21.15.116:0/4021458233 >> 172.21.15.116:0/4021458233 conn(0x7f11e406caf0 msgr2=0x7f11e406cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 -- 172.21.15.116:0/4021458233 shutdown_connections 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.612+0000 7f11eaaad640 1 -- 172.21.15.116:0/4021458233 wait complete. 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.613+0000 7f11eaaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.614+0000 7f11eaaad640 1 Processor -- start 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.614+0000 7f11eaaad640 1 -- start start 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.615+0000 7f11eaaad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.615+0000 7f11eaaad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11e4126be0 con 0x7f11e409fe30 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.615+0000 7f11e8849640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.615+0000 7f11e8849640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56478/0 (socket says 172.21.15.116:56478) 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.615+0000 7f11e8849640 1 -- 172.21.15.116:0/3802391851 learned_addr learned my addr 172.21.15.116:0/3802391851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11e8849640 1 -- 172.21.15.116:0/3802391851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e41251d0 con 0x7f11e409fe30 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11e8849640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f11d400eda0 con 0x7f11e409fe30 2024-08-16T02:42:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f11d4015420 con 0x7f11e409fe30 2024-08-16T02:42:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e4124e80 con 0x7f11e409fe30 2024-08-16T02:42:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f11d40156c0 con 0x7f11e409fe30 2024-08-16T02:42:12.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.616+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e4125eb0 con 0x7f11e409fe30 2024-08-16T02:42:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.617+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f11d4015870 con 0x7f11e409fe30 2024-08-16T02:42:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.617+0000 7f11e17fa640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 0x7f11d0040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.617+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e4125380 con 0x7f11e409fe30 2024-08-16T02:42:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.617+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f11d404fd90 con 0x7f11e409fe30 2024-08-16T02:42:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.618+0000 7f11e3fff640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 0x7f11d0040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.618+0000 7f11e3fff640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 0x7f11d0040220 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.623+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f11d400f2a0 con 0x7f11e409fe30 2024-08-16T02:42:12.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.802+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11e40a43b0 con 0x7f11d003dd10 2024-08-16T02:42:12.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.804+0000 7f11e17fa640 1 -- 172.21.15.116:0/3802391851 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f11d80012b0 con 0x7f11d003dd10 2024-08-16T02:42:12.805 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:12.805 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 msgr2=0x7f11d0040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 0x7f11d0040220 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 msgr2=0x7f11e4123540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 crc :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 shutdown_connections 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f11d003dd10 0x7f11d0040220 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 --2- 172.21.15.116:0/3802391851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e409fe30 0x7f11e4123540 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.806+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 >> 172.21.15.116:0/3802391851 conn(0x7f11e406caf0 msgr2=0x7f11e40715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:12.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.807+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 shutdown_connections 2024-08-16T02:42:12.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.807+0000 7f11eaaad640 1 -- 172.21.15.116:0/3802391851 wait complete. 2024-08-16T02:42:12.818 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:42:11.729008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16816,"kb_used_data":16816,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174096,"statfs":{"total":214211493888,"available":214194274304,"internally_reserved":0,"allocated":17219584,"data_stored":17219584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000923"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.358593+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:42:08.358593+0000","last_peered":"2024-08-16T02:42:08.358593+0000","last_clean":"2024-08-16T02:42:08.358593+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:42:08.358593+0000","last_undegraded":"2024-08-16T02:42:08.358593+0000","last_fullsized":"2024-08-16T02:42:08.358593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353195+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:42:08.353195+0000","last_peered":"2024-08-16T02:42:08.353195+0000","last_clean":"2024-08-16T02:42:08.353195+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:42:08.353195+0000","last_undegraded":"2024-08-16T02:42:08.353195+0000","last_fullsized":"2024-08-16T02:42:08.353195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353789+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:42:08.353789+0000","last_peered":"2024-08-16T02:42:08.353789+0000","last_clean":"2024-08-16T02:42:08.353789+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:42:08.353789+0000","last_undegraded":"2024-08-16T02:42:08.353789+0000","last_fullsized":"2024-08-16T02:42:08.353789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.324614+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:42:08.324614+0000","last_peered":"2024-08-16T02:42:08.324614+0000","last_clean":"2024-08-16T02:42:08.324614+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:42:08.324614+0000","last_undegraded":"2024-08-16T02:42:08.324614+0000","last_fullsized":"2024-08-16T02:42:08.324614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.340895+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:42:08.340895+0000","last_peered":"2024-08-16T02:42:08.340895+0000","last_clean":"2024-08-16T02:42:08.340895+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:42:08.340895+0000","last_undegraded":"2024-08-16T02:42:08.340895+0000","last_fullsized":"2024-08-16T02:42:08.340895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.328255+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:42:08.328255+0000","last_peered":"2024-08-16T02:42:08.328255+0000","last_clean":"2024-08-16T02:42:08.328255+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:42:08.328255+0000","last_undegraded":"2024-08-16T02:42:08.328255+0000","last_fullsized":"2024-08-16T02:42:08.328255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.357436+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:42:08.357436+0000","last_peered":"2024-08-16T02:42:08.357436+0000","last_clean":"2024-08-16T02:42:08.357436+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:42:08.357436+0000","last_undegraded":"2024-08-16T02:42:08.357436+0000","last_fullsized":"2024-08-16T02:42:08.357436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.361357+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:42:08.361357+0000","last_peered":"2024-08-16T02:42:08.361357+0000","last_clean":"2024-08-16T02:42:08.361357+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:42:08.361357+0000","last_undegraded":"2024-08-16T02:42:08.361357+0000","last_fullsized":"2024-08-16T02:42:08.361357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.323301+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:42:08.323301+0000","last_peered":"2024-08-16T02:42:08.323301+0000","last_clean":"2024-08-16T02:42:08.323301+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:42:08.323301+0000","last_undegraded":"2024-08-16T02:42:08.323301+0000","last_fullsized":"2024-08-16T02:42:08.323301+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198419456,"internally_reserved":0,"allocated":13074432,"data_stored":13074432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3688,"kb_used_data":3688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294040,"statfs":{"total":53552873472,"available":53549096960,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550002176,"internally_reserved":0,"allocated":2871296,"data_stored":2871296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:42:12.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:42:12.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.929+0000 7fde0f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:12.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.933+0000 7fde0f86d640 1 Processor -- start 2024-08-16T02:42:12.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.933+0000 7fde0f86d640 1 -- start start 2024-08-16T02:42:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.934+0000 7fde0f86d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde0809e340 0x7fde08138540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0f86d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde080a0f70 con 0x7fde0806f660 2024-08-16T02:42:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0f86d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fde080a0d00 con 0x7fde0809e340 2024-08-16T02:42:12.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0d609640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fde0806f660 0x7fde0809dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52270/0 (socket says 172.21.15.116:52270) 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0ce08640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde0809e340 0x7fde08138540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0d609640 1 -- 172.21.15.116:0/160776190 learned_addr learned my addr 172.21.15.116:0/160776190 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0ce08640 1 -- 172.21.15.116:0/160776190 >> v1:172.21.15.116:6789/0 conn(0x7fde0806f660 legacy=0x7fde0809dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0ce08640 1 -- 172.21.15.116:0/160776190 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde0809ee50 con 0x7fde0809e340 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde07fff640 1 -- 172.21.15.116:0/160776190 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3537723870 0 0) 0x7fde080a0f70 con 0x7fde0806f660 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde0ce08640 1 --2- 172.21.15.116:0/160776190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0809e340 0x7fde08138540 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9a0c878ef3c34a server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde07fff640 1 -- 172.21.15.116:0/160776190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fddfc009d90 con 0x7fde0809e340 2024-08-16T02:42:12.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde07fff640 1 -- 172.21.15.116:0/160776190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fddfc00e230 con 0x7fde0809e340 2024-08-16T02:42:12.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.935+0000 7fde07fff640 1 -- 172.21.15.116:0/160776190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fddfc00e4d0 con 0x7fde0809e340 2024-08-16T02:42:12.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 -- 172.21.15.116:0/160776190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0809e340 msgr2=0x7fde08138540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:12.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 --2- 172.21.15.116:0/160776190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0809e340 0x7fde08138540 crc :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 -- 172.21.15.116:0/160776190 shutdown_connections 2024-08-16T02:42:12.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 --2- 172.21.15.116:0/160776190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0809e340 0x7fde08138540 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:12.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 -- 172.21.15.116:0/160776190 >> 172.21.15.116:0/160776190 conn(0x7fde0809f1d0 msgr2=0x7fde0809f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:12.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 -- 172.21.15.116:0/160776190 shutdown_connections 2024-08-16T02:42:12.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 1 -- 172.21.15.116:0/160776190 wait complete. 2024-08-16T02:42:12.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.936+0000 7fde0f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:12.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.938+0000 7fde0f86d640 1 Processor -- start 2024-08-16T02:42:12.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.938+0000 7fde0f86d640 1 -- start start 2024-08-16T02:42:12.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0f86d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0f86d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde081208b0 con 0x7fde0806f660 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0d609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0d609640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56486/0 (socket says 172.21.15.116:56486) 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0d609640 1 -- 172.21.15.116:0/2615936182 learned_addr learned my addr 172.21.15.116:0/2615936182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0d609640 1 -- 172.21.15.116:0/2615936182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde08120e40 con 0x7fde0806f660 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0d609640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fddf8004b80 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fddf8004ce0 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde08122ca0 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.939+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fddf8007120 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.940+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde08120aa0 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.940+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fddf80072d0 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.940+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde0809dc30 con 0x7fde0806f660 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.941+0000 7fde05ffb640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 0x7fddf4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:12.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.941+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fddf804e860 con 0x7fde0806f660 2024-08-16T02:42:12.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.941+0000 7fde0ce08640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 0x7fddf4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:12.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.942+0000 7fde0ce08640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 0x7fddf4040220 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:12.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:12.946+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fddf800adb0 con 0x7fde0806f660 2024-08-16T02:42:13.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.127+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fde0806bf10 con 0x7fddf403dd10 2024-08-16T02:42:13.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.128+0000 7fde05ffb640 1 -- 172.21.15.116:0/2615936182 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7fde0806bf10 con 0x7fddf403dd10 2024-08-16T02:42:13.132 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:13.133 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:42:13.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 msgr2=0x7fddf4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 0x7fddf4040220 crc :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 msgr2=0x7fde08121390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 crc :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 shutdown_connections 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fddf403dd10 0x7fddf4040220 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 --2- 172.21.15.116:0/2615936182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde0806f660 0x7fde08121390 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 >> 172.21.15.116:0/2615936182 conn(0x7fde0809f1d0 msgr2=0x7fde080725d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:13.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.130+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 shutdown_connections 2024-08-16T02:42:13.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.131+0000 7fde0f86d640 1 -- 172.21.15.116:0/2615936182 wait complete. 2024-08-16T02:42:13.142 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:42:11.729008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16816,"kb_used_data":16816,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174096,"statfs":{"total":214211493888,"available":214194274304,"internally_reserved":0,"allocated":17219584,"data_stored":17219584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000923"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.358593+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:42:08.358593+0000","last_peered":"2024-08-16T02:42:08.358593+0000","last_clean":"2024-08-16T02:42:08.358593+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:42:08.358593+0000","last_undegraded":"2024-08-16T02:42:08.358593+0000","last_fullsized":"2024-08-16T02:42:08.358593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353195+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:42:08.353195+0000","last_peered":"2024-08-16T02:42:08.353195+0000","last_clean":"2024-08-16T02:42:08.353195+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:42:08.353195+0000","last_undegraded":"2024-08-16T02:42:08.353195+0000","last_fullsized":"2024-08-16T02:42:08.353195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353789+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:42:08.353789+0000","last_peered":"2024-08-16T02:42:08.353789+0000","last_clean":"2024-08-16T02:42:08.353789+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:42:08.353789+0000","last_undegraded":"2024-08-16T02:42:08.353789+0000","last_fullsized":"2024-08-16T02:42:08.353789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.324614+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:42:08.324614+0000","last_peered":"2024-08-16T02:42:08.324614+0000","last_clean":"2024-08-16T02:42:08.324614+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:42:08.324614+0000","last_undegraded":"2024-08-16T02:42:08.324614+0000","last_fullsized":"2024-08-16T02:42:08.324614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.340895+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:42:08.340895+0000","last_peered":"2024-08-16T02:42:08.340895+0000","last_clean":"2024-08-16T02:42:08.340895+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:42:08.340895+0000","last_undegraded":"2024-08-16T02:42:08.340895+0000","last_fullsized":"2024-08-16T02:42:08.340895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.328255+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:42:08.328255+0000","last_peered":"2024-08-16T02:42:08.328255+0000","last_clean":"2024-08-16T02:42:08.328255+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:42:08.328255+0000","last_undegraded":"2024-08-16T02:42:08.328255+0000","last_fullsized":"2024-08-16T02:42:08.328255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.357436+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:42:08.357436+0000","last_peered":"2024-08-16T02:42:08.357436+0000","last_clean":"2024-08-16T02:42:08.357436+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:42:08.357436+0000","last_undegraded":"2024-08-16T02:42:08.357436+0000","last_fullsized":"2024-08-16T02:42:08.357436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.361357+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:42:08.361357+0000","last_peered":"2024-08-16T02:42:08.361357+0000","last_clean":"2024-08-16T02:42:08.361357+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:42:08.361357+0000","last_undegraded":"2024-08-16T02:42:08.361357+0000","last_fullsized":"2024-08-16T02:42:08.361357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.323301+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:42:08.323301+0000","last_peered":"2024-08-16T02:42:08.323301+0000","last_clean":"2024-08-16T02:42:08.323301+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:42:08.323301+0000","last_undegraded":"2024-08-16T02:42:08.323301+0000","last_fullsized":"2024-08-16T02:42:08.323301+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198419456,"internally_reserved":0,"allocated":13074432,"data_stored":13074432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3688,"kb_used_data":3688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294040,"statfs":{"total":53552873472,"available":53549096960,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550002176,"internally_reserved":0,"allocated":2871296,"data_stored":2871296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:42:13.143 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-16T02:42:13.143 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:42:13.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.247+0000 7f3e1fb12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:13.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.252+0000 7f3e1fb12640 1 Processor -- start 2024-08-16T02:42:13.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.252+0000 7f3e1fb12640 1 -- start start 2024-08-16T02:42:13.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.253+0000 7f3e1fb12640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e181326c0 0x7f3e181371b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.253+0000 7f3e1fb12640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e1809dad0 con 0x7f3e1806cb20 2024-08-16T02:42:13.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.253+0000 7f3e1fb12640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3e18070580 con 0x7f3e181326c0 2024-08-16T02:42:13.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e181326c0 0x7f3e181371b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e181326c0 0x7f3e181371b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56490/0 (socket says 172.21.15.116:56490) 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d8ae640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e1806cb20 0x7f3e18131fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52278/0 (socket says 172.21.15.116:52278) 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 -- 172.21.15.116:0/3956681773 learned_addr learned my addr 172.21.15.116:0/3956681773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 -- 172.21.15.116:0/3956681773 >> v1:172.21.15.116:6789/0 conn(0x7f3e1806cb20 legacy=0x7f3e18131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 -- 172.21.15.116:0/3956681773 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1809dd20 con 0x7f3e181326c0 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1d0ad640 1 --2- 172.21.15.116:0/3956681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e181326c0 0x7f3e181371b0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624e5a3e82d5bea6 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1c8ac640 1 -- 172.21.15.116:0/3956681773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e08009e50 con 0x7f3e181326c0 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1c8ac640 1 -- 172.21.15.116:0/3956681773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3e0800e230 con 0x7f3e181326c0 2024-08-16T02:42:13.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.254+0000 7f3e1c8ac640 1 -- 172.21.15.116:0/3956681773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e0800e4d0 con 0x7f3e181326c0 2024-08-16T02:42:13.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 -- 172.21.15.116:0/3956681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e181326c0 msgr2=0x7f3e181371b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/3956681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e181326c0 0x7f3e181371b0 crc :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 -- 172.21.15.116:0/3956681773 shutdown_connections 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/3956681773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e181326c0 0x7f3e181371b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 -- 172.21.15.116:0/3956681773 >> 172.21.15.116:0/3956681773 conn(0x7f3e1806ec10 msgr2=0x7f3e1806f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 -- 172.21.15.116:0/3956681773 shutdown_connections 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 1 -- 172.21.15.116:0/3956681773 wait complete. 2024-08-16T02:42:13.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.255+0000 7f3e1fb12640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:13.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.257+0000 7f3e1fb12640 1 Processor -- start 2024-08-16T02:42:13.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.257+0000 7f3e1fb12640 1 -- start start 2024-08-16T02:42:13.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1fb12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1fb12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e18074530 con 0x7f3e1806cb20 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1d8ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1d8ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56504/0 (socket says 172.21.15.116:56504) 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1d8ae640 1 -- 172.21.15.116:0/2957618398 learned_addr learned my addr 172.21.15.116:0/2957618398 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1d8ae640 1 -- 172.21.15.116:0/2957618398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1809c180 con 0x7f3e1806cb20 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1d8ae640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e140059a0 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3e14005b00 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1811a490 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e14005da0 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.258+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e18074040 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.259+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f3e140070b0 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.259+0000 7f3e0e7fc640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 0x7f3dfc040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.259+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e1806a020 con 0x7f3e1806cb20 2024-08-16T02:42:13.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.260+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f3e1404e6a0 con 0x7f3e1806cb20 2024-08-16T02:42:13.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.260+0000 7f3e1d0ad640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 0x7f3dfc040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.261+0000 7f3e1d0ad640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 0x7f3dfc040220 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.265+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f3e1400de40 con 0x7f3e1806cb20 2024-08-16T02:42:13.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.448+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3e18132ce0 con 0x7f3dfc03dd10 2024-08-16T02:42:13.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.450+0000 7f3e0e7fc640 1 -- 172.21.15.116:0/2957618398 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7f3e0800e4a0 con 0x7f3dfc03dd10 2024-08-16T02:42:13.451 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:13.451 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:42:13.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 msgr2=0x7f3dfc040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 0x7f3dfc040220 crc :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 msgr2=0x7f3e18072dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 crc :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 shutdown_connections 2024-08-16T02:42:13.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3dfc03dd10 0x7f3dfc040220 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 --2- 172.21.15.116:0/2957618398 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e1806cb20 0x7f3e18072dc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 >> 172.21.15.116:0/2957618398 conn(0x7f3e1806ec10 msgr2=0x7f3e1812fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:13.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 shutdown_connections 2024-08-16T02:42:13.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.452+0000 7f3e1fb12640 1 -- 172.21.15.116:0/2957618398 wait complete. 2024-08-16T02:42:13.464 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":374,"stamp":"2024-08-16T02:42:11.729008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":16816,"kb_used_data":16816,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209174096,"statfs":{"total":214211493888,"available":214194274304,"internally_reserved":0,"allocated":17219584,"data_stored":17219584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000923"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.358593+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:42:08.358593+0000","last_peered":"2024-08-16T02:42:08.358593+0000","last_clean":"2024-08-16T02:42:08.358593+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:42:08.358593+0000","last_undegraded":"2024-08-16T02:42:08.358593+0000","last_fullsized":"2024-08-16T02:42:08.358593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353195+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:42:08.353195+0000","last_peered":"2024-08-16T02:42:08.353195+0000","last_clean":"2024-08-16T02:42:08.353195+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:42:08.353195+0000","last_undegraded":"2024-08-16T02:42:08.353195+0000","last_fullsized":"2024-08-16T02:42:08.353195+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.353789+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:42:08.353789+0000","last_peered":"2024-08-16T02:42:08.353789+0000","last_clean":"2024-08-16T02:42:08.353789+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:42:08.353789+0000","last_undegraded":"2024-08-16T02:42:08.353789+0000","last_fullsized":"2024-08-16T02:42:08.353789+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.324614+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:42:08.324614+0000","last_peered":"2024-08-16T02:42:08.324614+0000","last_clean":"2024-08-16T02:42:08.324614+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:42:08.324614+0000","last_undegraded":"2024-08-16T02:42:08.324614+0000","last_fullsized":"2024-08-16T02:42:08.324614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":647,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.340895+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:42:08.340895+0000","last_peered":"2024-08-16T02:42:08.340895+0000","last_clean":"2024-08-16T02:42:08.340895+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:42:08.340895+0000","last_undegraded":"2024-08-16T02:42:08.340895+0000","last_fullsized":"2024-08-16T02:42:08.340895+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.328255+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:42:08.328255+0000","last_peered":"2024-08-16T02:42:08.328255+0000","last_clean":"2024-08-16T02:42:08.328255+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:42:08.328255+0000","last_undegraded":"2024-08-16T02:42:08.328255+0000","last_fullsized":"2024-08-16T02:42:08.328255+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":645,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.357436+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:42:08.357436+0000","last_peered":"2024-08-16T02:42:08.357436+0000","last_clean":"2024-08-16T02:42:08.357436+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:42:08.357436+0000","last_undegraded":"2024-08-16T02:42:08.357436+0000","last_fullsized":"2024-08-16T02:42:08.357436+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":646,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.361357+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:42:08.361357+0000","last_peered":"2024-08-16T02:42:08.361357+0000","last_clean":"2024-08-16T02:42:08.361357+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:42:08.361357+0000","last_undegraded":"2024-08-16T02:42:08.361357+0000","last_fullsized":"2024-08-16T02:42:08.361357+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":717,"reported_epoch":225,"state":"active+clean","last_fresh":"2024-08-16T02:42:08.323301+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:42:08.323301+0000","last_peered":"2024-08-16T02:42:08.323301+0000","last_clean":"2024-08-16T02:42:08.323301+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:42:08.323301+0000","last_undegraded":"2024-08-16T02:42:08.323301+0000","last_fullsized":"2024-08-16T02:42:08.323301+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198419456,"internally_reserved":0,"allocated":13074432,"data_stored":13074432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4584,"kb_used_data":4584,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293144,"statfs":{"total":53552873472,"available":53548179456,"internally_reserved":0,"allocated":4694016,"data_stored":4694016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3688,"kb_used_data":3688,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294040,"statfs":{"total":53552873472,"available":53549096960,"internally_reserved":0,"allocated":3776512,"data_stored":3776512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3684,"kb_used_data":3684,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294044,"statfs":{"total":53552873472,"available":53549101056,"internally_reserved":0,"allocated":3772416,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":4860,"kb_used_data":4860,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52292868,"statfs":{"total":53552873472,"available":53547896832,"internally_reserved":0,"allocated":4976640,"data_stored":4976640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549064192,"internally_reserved":0,"allocated":3809280,"data_stored":3809280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53550006272,"internally_reserved":0,"allocated":2867200,"data_stored":2867200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53550002176,"internally_reserved":0,"allocated":2871296,"data_stored":2871296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549346816,"internally_reserved":0,"allocated":3526656,"data_stored":3526656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:42:13.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-16T02:42:13.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.567+0000 7f92c9024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:13.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.572+0000 7f92c9024640 1 Processor -- start 2024-08-16T02:42:13.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.572+0000 7f92c9024640 1 -- start start 2024-08-16T02:42:13.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92c9024640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92c4135150 0x7f92c4137570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92c9024640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92c409ca30 con 0x7f92c41322e0 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92c9024640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f92c409e6b0 con 0x7f92c4135150 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92bbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92c4135150 0x7f92c4137570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92bbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92c4135150 0x7f92c4137570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56518/0 (socket says 172.21.15.116:56518) 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92bbfff640 1 -- 172.21.15.116:0/3537550516 learned_addr learned my addr 172.21.15.116:0/3537550516 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:13.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92c2d9d640 1 --1- 172.21.15.116:0/3537550516 >> v1:172.21.15.116:6789/0 conn(0x7f92c41322e0 0x7f92c41326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52284/0 (socket says 172.21.15.116:52284) 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92bbfff640 1 -- 172.21.15.116:0/3537550516 >> v1:172.21.15.116:6789/0 conn(0x7f92c41322e0 legacy=0x7f92c41326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.573+0000 7f92bbfff640 1 -- 172.21.15.116:0/3537550516 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c409e3a0 con 0x7f92c4135150 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92bbfff640 1 --2- 172.21.15.116:0/3537550516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c4135150 0x7f92c4137570 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b4722004912255 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92c259c640 1 -- 172.21.15.116:0/3537550516 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f92ac009e50 con 0x7f92c4135150 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92c259c640 1 -- 172.21.15.116:0/3537550516 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f92ac00e230 con 0x7f92c4135150 2024-08-16T02:42:13.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92c259c640 1 -- 172.21.15.116:0/3537550516 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f92ac00e4d0 con 0x7f92c4135150 2024-08-16T02:42:13.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92c9024640 1 -- 172.21.15.116:0/3537550516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c4135150 msgr2=0x7f92c4137570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.574+0000 7f92c9024640 1 --2- 172.21.15.116:0/3537550516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c4135150 0x7f92c4137570 crc :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 1 -- 172.21.15.116:0/3537550516 shutdown_connections 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 1 --2- 172.21.15.116:0/3537550516 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c4135150 0x7f92c4137570 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 1 -- 172.21.15.116:0/3537550516 >> 172.21.15.116:0/3537550516 conn(0x7f92c409f1d0 msgr2=0x7f92c409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 1 -- 172.21.15.116:0/3537550516 shutdown_connections 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 1 -- 172.21.15.116:0/3537550516 wait complete. 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.575+0000 7f92c9024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:13.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.576+0000 7f92c9024640 1 Processor -- start 2024-08-16T02:42:13.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.576+0000 7f92c9024640 1 -- start start 2024-08-16T02:42:13.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.577+0000 7f92c9024640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.577+0000 7f92c9024640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f92c4124e80 con 0x7f92c41322e0 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.577+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.577+0000 7f92c2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56520/0 (socket says 172.21.15.116:56520) 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.577+0000 7f92c2d9d640 1 -- 172.21.15.116:0/812915914 learned_addr learned my addr 172.21.15.116:0/812915914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92c2d9d640 1 -- 172.21.15.116:0/812915914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92c409def0 con 0x7f92c41322e0 2024-08-16T02:42:13.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92c2d9d640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f92b40059a0 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f92b4005b00 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92c411f320 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f92b4005da0 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.578+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92c4124210 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.579+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f92b40070b0 con 0x7f92c41322e0 2024-08-16T02:42:13.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.579+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92c40a37d0 con 0x7f92c41322e0 2024-08-16T02:42:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.579+0000 7f92bb7fe640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 0x7f92a4040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.579+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f92b404e4d0 con 0x7f92c41322e0 2024-08-16T02:42:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.580+0000 7f92bbfff640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 0x7f92a4040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:13.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.580+0000 7f92bbfff640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 0x7f92a4040220 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:13.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.584+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f92b400de10 con 0x7f92c41322e0 2024-08-16T02:42:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.777+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92c409def0 con 0x7f92c41322e0 2024-08-16T02:42:13.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.778+0000 7f92bb7fe640 1 -- 172.21.15.116:0/812915914 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) ==== 74+0+8854 (crc 0 0 0) 0x7f92c409def0 con 0x7f92c41322e0 2024-08-16T02:42:13.779 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:13.779 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":228,"fsid":"8b1d1868-3d02-4f61-802a-791ead3a36d9","created":"2024-08-16T02:36:57.167404+0000","modified":"2024-08-16T02:42:11.299140+0000","last_up_change":"2024-08-16T02:37:03.718029+0000","last_in_change":"2024-08-16T02:36:59.302189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":72,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-16T02:37:05.689624+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-16T02:37:07.533370+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"18f869dd-54a7-4f0a-bd69-be3b5fd67da7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":295590560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6803","nonce":295590560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":295590560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6805","nonce":295590560}]},"public_addr":"172.21.15.116:6804/295590560","cluster_addr":"172.21.15.116:6803/295590560","heartbeat_back_addr":"172.21.15.116:6806/295590560","heartbeat_front_addr":"172.21.15.116:6805/295590560","state":["exists","up"]},{"osd":1,"uuid":"d5599bff-f9e3-4704-a9c2-f90504977470","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":2086482905}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6801","nonce":2086482905}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6807","nonce":2086482905}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":2086482905}]},"public_addr":"172.21.15.116:6802/2086482905","cluster_addr":"172.21.15.116:6801/2086482905","heartbeat_back_addr":"172.21.15.116:6807/2086482905","heartbeat_front_addr":"172.21.15.116:6808/2086482905","state":["exists","up"]},{"osd":2,"uuid":"08298ee2-d217-4571-847a-605bd52c18d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6820","nonce":1500775361}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6821","nonce":1500775361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":1500775361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":1500775361}]},"public_addr":"172.21.15.152:6820/1500775361","cluster_addr":"172.21.15.152:6821/1500775361","heartbeat_back_addr":"172.21.15.152:6806/1500775361","heartbeat_front_addr":"172.21.15.152:6807/1500775361","state":["exists","up"]},{"osd":3,"uuid":"f095864e-6d26-4aeb-9a17-b49b16ab8210","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":226,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2721550083}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2721550083}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2721550083}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2721550083}]},"public_addr":"172.21.15.152:6804/2721550083","cluster_addr":"172.21.15.152:6803/2721550083","heartbeat_back_addr":"172.21.15.152:6805/2721550083","heartbeat_front_addr":"172.21.15.152:6802/2721550083","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-16T02:42:13.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 msgr2=0x7f92a4040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 0x7f92a4040220 crc :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 msgr2=0x7f92c411db20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 crc :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 shutdown_connections 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f92a403dd10 0x7f92a4040220 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 --2- 172.21.15.116:0/812915914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92c41322e0 0x7f92c411db20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 >> 172.21.15.116:0/812915914 conn(0x7f92c409f1d0 msgr2=0x7f92c4132f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:13.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 shutdown_connections 2024-08-16T02:42:13.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:13.780+0000 7f92c9024640 1 -- 172.21.15.116:0/812915914 wait complete. 2024-08-16T02:42:14.794 INFO:tasks.ceph:Scrubbing osd.0 2024-08-16T02:42:14.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:42:14.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.912+0000 7faa2bc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.921+0000 7faa2bc2d640 1 Processor -- start 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.922+0000 7faa2bc2d640 1 -- start start 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa2bc2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa24071eb0 0x7faa2413a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa2bc2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa24072ce0 con 0x7faa24071390 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa2bc2d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faa240733e0 con 0x7faa24071eb0 2024-08-16T02:42:14.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa291c8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa24071eb0 0x7faa2413a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa299c9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa24071390 0x7faa240717a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52286/0 (socket says 172.21.15.116:52286) 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa291c8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa24071eb0 0x7faa2413a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56528/0 (socket says 172.21.15.116:56528) 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa299c9640 1 -- 172.21.15.116:0/2323879381 learned_addr learned my addr 172.21.15.116:0/2323879381 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa291c8640 1 -- 172.21.15.116:0/2323879381 >> v1:172.21.15.116:6789/0 conn(0x7faa24071390 legacy=0x7faa240717a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa291c8640 1 -- 172.21.15.116:0/2323879381 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24073f80 con 0x7faa24071eb0 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa289c7640 1 -- 172.21.15.116:0/2323879381 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571994803 0 0) 0x7faa24072ce0 con 0x7faa24071390 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.923+0000 7faa291c8640 1 --2- 172.21.15.116:0/2323879381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071eb0 0x7faa2413a670 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e25c81b4c5aa0a server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:14.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa289c7640 1 -- 172.21.15.116:0/2323879381 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faa18009d80 con 0x7faa24071eb0 2024-08-16T02:42:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa289c7640 1 -- 172.21.15.116:0/2323879381 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faa1800e230 con 0x7faa24071eb0 2024-08-16T02:42:14.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa289c7640 1 -- 172.21.15.116:0/2323879381 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faa1800e4d0 con 0x7faa24071eb0 2024-08-16T02:42:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2323879381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071eb0 msgr2=0x7faa2413a670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2323879381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071eb0 0x7faa2413a670 crc :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2323879381 shutdown_connections 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2323879381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071eb0 0x7faa2413a670 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.924+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2323879381 >> 172.21.15.116:0/2323879381 conn(0x7faa240a58f0 msgr2=0x7faa240a5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.925+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2323879381 shutdown_connections 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.925+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2323879381 wait complete. 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.925+0000 7faa2bc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.926+0000 7faa2bc2d640 1 Processor -- start 2024-08-16T02:42:14.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.926+0000 7faa2bc2d640 1 -- start start 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.927+0000 7faa2bc2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.927+0000 7faa2bc2d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa24122b30 con 0x7faa24071390 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.927+0000 7faa299c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.927+0000 7faa299c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56530/0 (socket says 172.21.15.116:56530) 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.927+0000 7faa299c9640 1 -- 172.21.15.116:0/2338222172 learned_addr learned my addr 172.21.15.116:0/2338222172 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa299c9640 1 -- 172.21.15.116:0/2338222172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa241230c0 con 0x7faa24071390 2024-08-16T02:42:14.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa299c9640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faa14004b80 con 0x7faa24071390 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faa14004ce0 con 0x7faa24071390 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa24124f20 con 0x7faa24071390 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faa14007120 con 0x7faa24071390 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.928+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa24122d20 con 0x7faa24071390 2024-08-16T02:42:14.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa24073c70 con 0x7faa24071390 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7faa140072d0 con 0x7faa24071390 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 0x7faa10040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7faa1404e860 con 0x7faa24071390 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 --2- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 0x7faa100451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 --> v2:172.21.15.116:6804/295590560 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa10045850 con 0x7faa10042d10 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.929+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7faa1404ebf0 con 0x7faa24071390 2024-08-16T02:42:14.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.930+0000 7faa291c8640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 0x7faa10040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.930+0000 7faa291c8640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 0x7faa10040220 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:14.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.930+0000 7faa2a1ca640 1 --2- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 0x7faa100451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:42:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.934+0000 7faa2a1ca640 1 --2- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 0x7faa100451c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:14.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.935+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== osd.0 v2:172.21.15.116:6804/295590560 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7faa10045850 con 0x7faa10042d10 2024-08-16T02:42:14.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.939+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 --> v2:172.21.15.116:6804/295590560 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faa24073960 con 0x7faa10042d10 2024-08-16T02:42:14.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa227fc640 1 -- 172.21.15.116:0/2338222172 <== osd.0 v2:172.21.15.116:6804/295590560 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7faa1c00a820 con 0x7faa10042d10 2024-08-16T02:42:14.941 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-16T02:42:14.941 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "" 2024-08-16T02:42:14.941 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 msgr2=0x7faa100451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 0x7faa100451c0 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 msgr2=0x7faa10040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 0x7faa10040220 crc :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 msgr2=0x7faa24123610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:14.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.940+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 crc :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 shutdown_connections 2024-08-16T02:42:14.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> v2:172.21.15.116:6804/295590560 conn(0x7faa10042d10 0x7faa100451c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faa1003dd10 0x7faa10040220 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 --2- 172.21.15.116:0/2338222172 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa24071390 0x7faa24123610 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:14.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 >> 172.21.15.116:0/2338222172 conn(0x7faa240a58f0 msgr2=0x7faa2406fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:14.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 shutdown_connections 2024-08-16T02:42:14.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:14.941+0000 7faa2bc2d640 1 -- 172.21.15.116:0/2338222172 wait complete. 2024-08-16T02:42:14.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-08-16T02:42:15.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.055+0000 7f698ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.061+0000 7f698ec2d640 1 Processor -- start 2024-08-16T02:42:15.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.061+0000 7f698ec2d640 1 -- start start 2024-08-16T02:42:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f698ec2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6988132cc0 0x7f69881376d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f698ec2d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f698809dc90 con 0x7f69881321a0 2024-08-16T02:42:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f698ec2d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f698809eab0 con 0x7f6988132cc0 2024-08-16T02:42:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f698c9c9640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f69881321a0 0x7f69881325b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52300/0 (socket says 172.21.15.116:52300) 2024-08-16T02:42:15.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f698c9c9640 1 -- 172.21.15.116:0/2488486720 learned_addr learned my addr 172.21.15.116:0/2488486720 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:42:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f6987fff640 1 --2- 172.21.15.116:0/2488486720 >> v2:172.21.15.116:3300/0 conn(0x7f6988132cc0 0x7f69881376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.062+0000 7f6987fff640 1 -- 172.21.15.116:0/2488486720 >> v1:172.21.15.116:6789/0 conn(0x7f69881321a0 legacy=0x7f69881325b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f6987fff640 1 -- 172.21.15.116:0/2488486720 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698806d990 con 0x7f6988132cc0 2024-08-16T02:42:15.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f69877fe640 1 -- 172.21.15.116:0/2488486720 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642813367 0 0) 0x7f698809dc90 con 0x7f69881321a0 2024-08-16T02:42:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f6987fff640 1 --2- 172.21.15.116:0/2488486720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6988132cc0 0x7f69881376d0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d214fa95764e57b2 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f69877fe640 1 -- 172.21.15.116:0/2488486720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f6974009d80 con 0x7f6988132cc0 2024-08-16T02:42:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f69877fe640 1 -- 172.21.15.116:0/2488486720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f697400e230 con 0x7f6988132cc0 2024-08-16T02:42:15.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f69877fe640 1 -- 172.21.15.116:0/2488486720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f697400e4d0 con 0x7f6988132cc0 2024-08-16T02:42:15.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f698ec2d640 1 -- 172.21.15.116:0/2488486720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6988132cc0 msgr2=0x7f69881376d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.063+0000 7f698ec2d640 1 --2- 172.21.15.116:0/2488486720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6988132cc0 0x7f69881376d0 crc :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 1 -- 172.21.15.116:0/2488486720 shutdown_connections 2024-08-16T02:42:15.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 1 --2- 172.21.15.116:0/2488486720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6988132cc0 0x7f69881376d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 1 -- 172.21.15.116:0/2488486720 >> 172.21.15.116:0/2488486720 conn(0x7f698809f1d0 msgr2=0x7f698809f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 1 -- 172.21.15.116:0/2488486720 shutdown_connections 2024-08-16T02:42:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 1 -- 172.21.15.116:0/2488486720 wait complete. 2024-08-16T02:42:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.064+0000 7f698ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.065+0000 7f698ec2d640 1 Processor -- start 2024-08-16T02:42:15.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.065+0000 7f698ec2d640 1 -- start start 2024-08-16T02:42:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.066+0000 7f698ec2d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.066+0000 7f698ec2d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f698811e660 con 0x7f69881321a0 2024-08-16T02:42:15.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.066+0000 7f698c9c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.066+0000 7f698c9c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56550/0 (socket says 172.21.15.116:56550) 2024-08-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.066+0000 7f698c9c9640 1 -- 172.21.15.116:0/1238766215 learned_addr learned my addr 172.21.15.116:0/1238766215 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f698c9c9640 1 -- 172.21.15.116:0/1238766215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f698809dec0 con 0x7f69881321a0 2024-08-16T02:42:15.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f698c9c9640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f697c004b80 con 0x7f69881321a0 2024-08-16T02:42:15.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f698811db40 con 0x7f69881321a0 2024-08-16T02:42:15.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f697c004ce0 con 0x7f69881321a0 2024-08-16T02:42:15.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f697c007120 con 0x7f69881321a0 2024-08-16T02:42:15.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.067+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f698812fca0 con 0x7f69881321a0 2024-08-16T02:42:15.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.068+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f697c007320 con 0x7f69881321a0 2024-08-16T02:42:15.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.068+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69881325b0 con 0x7f69881321a0 2024-08-16T02:42:15.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.068+0000 7f69857fa640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 0x7f696c0401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.068+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f697c04e8b0 con 0x7f69881321a0 2024-08-16T02:42:15.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.069+0000 7f6987fff640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 0x7f696c0401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.069+0000 7f6987fff640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 0x7f696c0401d0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.073+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f697c00adb0 con 0x7f69881321a0 2024-08-16T02:42:15.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.264+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f6988074670 con 0x7f696c03dcc0 2024-08-16T02:42:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.264+0000 7f69857fa640 1 -- 172.21.15.116:0/1238766215 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f697400e3a0 con 0x7f696c03dcc0 2024-08-16T02:42:15.265 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 0 to deep-scrub 2024-08-16T02:42:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 msgr2=0x7f696c0401d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 0x7f696c0401d0 crc :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 msgr2=0x7f698811adb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 crc :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 shutdown_connections 2024-08-16T02:42:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f696c03dcc0 0x7f696c0401d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 --2- 172.21.15.116:0/1238766215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69881321a0 0x7f698811adb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 >> 172.21.15.116:0/1238766215 conn(0x7f698809f1d0 msgr2=0x7f6988130110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 shutdown_connections 2024-08-16T02:42:15.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.268+0000 7f698ec2d640 1 -- 172.21.15.116:0/1238766215 wait complete. 2024-08-16T02:42:15.285 INFO:tasks.ceph:Scrubbing osd.1 2024-08-16T02:42:15.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:42:15.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.388+0000 7fe12da24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.398+0000 7fe12da24640 1 Processor -- start 2024-08-16T02:42:15.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.398+0000 7fe12da24640 1 -- start start 2024-08-16T02:42:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe12da24640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe128135800 0x7fe128135c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe12da24640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe128072d00 con 0x7fe12809fe30 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe12da24640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe128072060 con 0x7fe128135800 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe126ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe128135800 0x7fe128135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe11e7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe12809fe30 0x7fe1280a0240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52310/0 (socket says 172.21.15.116:52310) 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe11e7fc640 1 -- 172.21.15.116:0/876032385 learned_addr learned my addr 172.21.15.116:0/876032385 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe126ffd640 1 -- 172.21.15.116:0/876032385 >> v1:172.21.15.116:6789/0 conn(0x7fe12809fe30 legacy=0x7fe1280a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe1267fc640 1 -- 172.21.15.116:0/876032385 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276195762 0 0) 0x7fe128072d00 con 0x7fe12809fe30 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe126ffd640 1 -- 172.21.15.116:0/876032385 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe128135320 con 0x7fe128135800 2024-08-16T02:42:15.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.399+0000 7fe126ffd640 1 --2- 172.21.15.116:0/876032385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128135800 0x7fe128135c10 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5d552b9593a5a9 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe1267fc640 1 -- 172.21.15.116:0/876032385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe118009d90 con 0x7fe128135800 2024-08-16T02:42:15.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe1267fc640 1 -- 172.21.15.116:0/876032385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe11800e230 con 0x7fe128135800 2024-08-16T02:42:15.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe1267fc640 1 -- 172.21.15.116:0/876032385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe11800e4d0 con 0x7fe128135800 2024-08-16T02:42:15.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 -- 172.21.15.116:0/876032385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128135800 msgr2=0x7fe128135c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 --2- 172.21.15.116:0/876032385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128135800 0x7fe128135c10 crc :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 -- 172.21.15.116:0/876032385 shutdown_connections 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 --2- 172.21.15.116:0/876032385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128135800 0x7fe128135c10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 -- 172.21.15.116:0/876032385 >> 172.21.15.116:0/876032385 conn(0x7fe12806caf0 msgr2=0x7fe12806cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 -- 172.21.15.116:0/876032385 shutdown_connections 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.400+0000 7fe12da24640 1 -- 172.21.15.116:0/876032385 wait complete. 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.401+0000 7fe12da24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.402+0000 7fe12da24640 1 Processor -- start 2024-08-16T02:42:15.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.402+0000 7fe12da24640 1 -- start start 2024-08-16T02:42:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe12da24640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe12da24640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe12811f3f0 con 0x7fe12809fe30 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe126ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe126ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56566/0 (socket says 172.21.15.116:56566) 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe126ffd640 1 -- 172.21.15.116:0/2645637878 learned_addr learned my addr 172.21.15.116:0/2645637878 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.403+0000 7fe126ffd640 1 -- 172.21.15.116:0/2645637878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe12811ee40 con 0x7fe12809fe30 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe126ffd640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe118014540 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fe1180146a0 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe12811eb30 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fe118014940 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.404+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe128120210 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe1280728d0 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fe118014af0 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 0x7fe108040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fe118019040 con 0x7fe12809fe30 2024-08-16T02:42:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 --2- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 0x7fe1080451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 --> v2:172.21.15.116:6802/2086482905 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe108045850 con 0x7fe108042d10 2024-08-16T02:42:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.405+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7fe118052450 con 0x7fe12809fe30 2024-08-16T02:42:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.406+0000 7fe11e7fc640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 0x7fe108040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.406+0000 7fe1277fe640 1 --2- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 0x7fe1080451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:42:15.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.406+0000 7fe11e7fc640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 0x7fe108040220 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.409+0000 7fe1277fe640 1 --2- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 0x7fe1080451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.411+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== osd.1 v2:172.21.15.116:6802/2086482905 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fe108045850 con 0x7fe108042d10 2024-08-16T02:42:15.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.414+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 --> v2:172.21.15.116:6802/2086482905 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe1281204c0 con 0x7fe108042d10 2024-08-16T02:42:15.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe124ff9640 1 -- 172.21.15.116:0/2645637878 <== osd.1 v2:172.21.15.116:6802/2086482905 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7fe1281204c0 con 0x7fe108042d10 2024-08-16T02:42:15.417 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-16T02:42:15.417 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "" 2024-08-16T02:42:15.417 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-16T02:42:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 msgr2=0x7fe1080451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 0x7fe1080451c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 msgr2=0x7fe108040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 0x7fe108040220 crc :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 msgr2=0x7fe128131a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 crc :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 shutdown_connections 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> v2:172.21.15.116:6802/2086482905 conn(0x7fe108042d10 0x7fe1080451c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fe10803dd10 0x7fe108040220 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 --2- 172.21.15.116:0/2645637878 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe12809fe30 0x7fe128131a20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 >> 172.21.15.116:0/2645637878 conn(0x7fe12806caf0 msgr2=0x7fe128072f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 shutdown_connections 2024-08-16T02:42:15.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.416+0000 7fe12da24640 1 -- 172.21.15.116:0/2645637878 wait complete. 2024-08-16T02:42:15.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-08-16T02:42:15.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.531+0000 7f30d546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.537+0000 7f30d546d640 1 Processor -- start 2024-08-16T02:42:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.538+0000 7f30d546d640 1 -- start start 2024-08-16T02:42:15.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30d546d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30d0135570 0x7f30d0135980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30d546d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d006fba0 con 0x7f30d00a1e40 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30d546d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f30d00a2f30 con 0x7f30d0135570 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30d0135570 0x7f30d0135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f30d0135570 0x7f30d0135980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56568/0 (socket says 172.21.15.116:56568) 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 -- 172.21.15.116:0/3738461152 learned_addr learned my addr 172.21.15.116:0/3738461152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30c7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f30d00a1e40 0x7f30d00a2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52326/0 (socket says 172.21.15.116:52326) 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 -- 172.21.15.116:0/3738461152 >> v1:172.21.15.116:6789/0 conn(0x7f30d00a1e40 legacy=0x7f30d00a2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ce7fc640 1 -- 172.21.15.116:0/3738461152 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578327888 0 0) 0x7f30d006fba0 con 0x7f30d00a1e40 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 -- 172.21.15.116:0/3738461152 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d00a3920 con 0x7f30d0135570 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ceffd640 1 --2- 172.21.15.116:0/3738461152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0135570 0x7f30d0135980 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1282fe1e621924d server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ce7fc640 1 -- 172.21.15.116:0/3738461152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f30c0009e50 con 0x7f30d0135570 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.539+0000 7f30ce7fc640 1 -- 172.21.15.116:0/3738461152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f30c000e230 con 0x7f30d0135570 2024-08-16T02:42:15.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30ce7fc640 1 -- 172.21.15.116:0/3738461152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f30c000e4d0 con 0x7f30d0135570 2024-08-16T02:42:15.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 -- 172.21.15.116:0/3738461152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0135570 msgr2=0x7f30d0135980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 --2- 172.21.15.116:0/3738461152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0135570 0x7f30d0135980 crc :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 -- 172.21.15.116:0/3738461152 shutdown_connections 2024-08-16T02:42:15.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 --2- 172.21.15.116:0/3738461152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d0135570 0x7f30d0135980 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 -- 172.21.15.116:0/3738461152 >> 172.21.15.116:0/3738461152 conn(0x7f30d00a5960 msgr2=0x7f30d00a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 -- 172.21.15.116:0/3738461152 shutdown_connections 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.540+0000 7f30d546d640 1 -- 172.21.15.116:0/3738461152 wait complete. 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.541+0000 7f30d546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.542+0000 7f30d546d640 1 Processor -- start 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.542+0000 7f30d546d640 1 -- start start 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30d546d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30d546d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30d0124b20 con 0x7f30d00a1e40 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30ceffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56574/0 (socket says 172.21.15.116:56574) 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30ceffd640 1 -- 172.21.15.116:0/1630722734 learned_addr learned my addr 172.21.15.116:0/1630722734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.543+0000 7f30ceffd640 1 -- 172.21.15.116:0/1630722734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d0122a40 con 0x7f30d00a1e40 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30ceffd640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f30c000ee20 con 0x7f30d00a1e40 2024-08-16T02:42:15.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f30c0015420 con 0x7f30d00a1e40 2024-08-16T02:42:15.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f30c00156c0 con 0x7f30d00a1e40 2024-08-16T02:42:15.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d0124830 con 0x7f30d00a1e40 2024-08-16T02:42:15.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.544+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d0124ea0 con 0x7f30d00a1e40 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.545+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f30c0015870 con 0x7f30d00a1e40 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.545+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d01237d0 con 0x7f30d00a1e40 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.545+0000 7f30c77fe640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 0x7f30b0040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.545+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f30c004fd00 con 0x7f30d00a1e40 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.546+0000 7f30c7fff640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 0x7f30b0040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.546+0000 7f30c7fff640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 0x7f30b0040220 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.551+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f30c0012bc0 con 0x7f30d00a1e40 2024-08-16T02:42:15.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.734+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f30d006f420 con 0x7f30b003dd10 2024-08-16T02:42:15.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.735+0000 7f30c77fe640 1 -- 172.21.15.116:0/1630722734 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f30b80070b0 con 0x7f30b003dd10 2024-08-16T02:42:15.736 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 1 to deep-scrub 2024-08-16T02:42:15.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 msgr2=0x7f30b0040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 0x7f30b0040220 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 msgr2=0x7f30d01240f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 crc :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 shutdown_connections 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f30b003dd10 0x7f30b0040220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 --2- 172.21.15.116:0/1630722734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d00a1e40 0x7f30d01240f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 >> 172.21.15.116:0/1630722734 conn(0x7f30d00a5960 msgr2=0x7f30d009fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 shutdown_connections 2024-08-16T02:42:15.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.738+0000 7f30d546d640 1 -- 172.21.15.116:0/1630722734 wait complete. 2024-08-16T02:42:15.758 INFO:tasks.ceph:Scrubbing osd.2 2024-08-16T02:42:15.759 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:42:15.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.863+0000 7faeee192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.872+0000 7faeee192640 1 Processor -- start 2024-08-16T02:42:15.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.873+0000 7faeee192640 1 -- start start 2024-08-16T02:42:15.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faeee192640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faee809c540 0x7faee81383f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faeee192640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faee8071aa0 con 0x7faee809ba20 2024-08-16T02:42:15.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faeee192640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faee809df20 con 0x7faee809c540 2024-08-16T02:42:15.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faee809c540 0x7faee81383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faee809c540 0x7faee81383f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56580/0 (socket says 172.21.15.116:56580) 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faee809ba20 0x7faee809be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52328/0 (socket says 172.21.15.116:52328) 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 -- 172.21.15.116:0/393344233 learned_addr learned my addr 172.21.15.116:0/393344233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 -- 172.21.15.116:0/393344233 >> v1:172.21.15.116:6789/0 conn(0x7faee809ba20 legacy=0x7faee809be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 -- 172.21.15.116:0/393344233 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee809cb50 con 0x7faee809c540 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee6ffd640 1 --2- 172.21.15.116:0/393344233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809c540 0x7faee81383f0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd31165acea4dd27 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee67fc640 1 -- 172.21.15.116:0/393344233 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faedc009e50 con 0x7faee809c540 2024-08-16T02:42:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee67fc640 1 -- 172.21.15.116:0/393344233 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faedc00e230 con 0x7faee809c540 2024-08-16T02:42:15.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.874+0000 7faee67fc640 1 -- 172.21.15.116:0/393344233 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faedc00e4d0 con 0x7faee809c540 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 -- 172.21.15.116:0/393344233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809c540 msgr2=0x7faee81383f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 --2- 172.21.15.116:0/393344233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809c540 0x7faee81383f0 crc :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 -- 172.21.15.116:0/393344233 shutdown_connections 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 --2- 172.21.15.116:0/393344233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809c540 0x7faee81383f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 -- 172.21.15.116:0/393344233 >> 172.21.15.116:0/393344233 conn(0x7faee80a18d0 msgr2=0x7faee80a1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 -- 172.21.15.116:0/393344233 shutdown_connections 2024-08-16T02:42:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 1 -- 172.21.15.116:0/393344233 wait complete. 2024-08-16T02:42:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.875+0000 7faeee192640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.877+0000 7faeee192640 1 Processor -- start 2024-08-16T02:42:15.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.877+0000 7faeee192640 1 -- start start 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faeee192640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faeee192640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faee8122720 con 0x7faee809ba20 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faee77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faee77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56582/0 (socket says 172.21.15.116:56582) 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faee77fe640 1 -- 172.21.15.116:0/855369759 learned_addr learned my addr 172.21.15.116:0/855369759 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.878+0000 7faee77fe640 1 -- 172.21.15.116:0/855369759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faee8133ff0 con 0x7faee809ba20 2024-08-16T02:42:15.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faee77fe640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faed80071e0 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faed8007340 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faed80075e0 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faee8121f00 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.879+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faee8123540 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faee806f110 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7faed8007790 con 0x7faee809ba20 2024-08-16T02:42:15.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 0x7faed4040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7faed804f370 con 0x7faee809ba20 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 --2- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 0x7faed40451f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 --> v2:172.21.15.152:6820/1500775361 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faed4045880 con 0x7faed4042d40 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7faed8051860 con 0x7faee809ba20 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.880+0000 7faee6ffd640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 0x7faed4040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:15.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.881+0000 7faee6ffd640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 0x7faed4040250 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.882+0000 7faee7fff640 1 --2- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 0x7faed40451f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:42:15.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.886+0000 7faee7fff640 1 --2- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 0x7faed40451f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:15.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.888+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== osd.2 v2:172.21.15.152:6820/1500775361 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7faed4045880 con 0x7faed4042d40 2024-08-16T02:42:15.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.891+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 --> v2:172.21.15.152:6820/1500775361 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7faee809d0e0 con 0x7faed4042d40 2024-08-16T02:42:15.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.893+0000 7faecffff640 1 -- 172.21.15.116:0/855369759 <== osd.2 v2:172.21.15.152:6820/1500775361 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7faee0009830 con 0x7faed4042d40 2024-08-16T02:42:15.894 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-16T02:42:15.895 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "" 2024-08-16T02:42:15.895 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-16T02:42:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 msgr2=0x7faed40451f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 0x7faed40451f0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 msgr2=0x7faed4040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 0x7faed4040250 crc :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 msgr2=0x7faee8120d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 crc :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 shutdown_connections 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> v2:172.21.15.152:6820/1500775361 conn(0x7faed4042d40 0x7faed40451f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faed403dd40 0x7faed4040250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 --2- 172.21.15.116:0/855369759 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faee809ba20 0x7faee8120d70 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 >> 172.21.15.116:0/855369759 conn(0x7faee80a18d0 msgr2=0x7faee81ba070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 shutdown_connections 2024-08-16T02:42:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:15.894+0000 7faeee192640 1 -- 172.21.15.116:0/855369759 wait complete. 2024-08-16T02:42:15.906 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-08-16T02:42:16.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.009+0000 7faefaf52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.015+0000 7faefaf52640 1 Processor -- start 2024-08-16T02:42:16.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.015+0000 7faefaf52640 1 -- start start 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faefaf52640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faef41321a0 0x7faef41325b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faefaf52640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faef409dc90 con 0x7faef4132af0 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faefaf52640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faef409eab0 con 0x7faef41321a0 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faef8cee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faef41321a0 0x7faef41325b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faef8cee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faef41321a0 0x7faef41325b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56590/0 (socket says 172.21.15.116:56590) 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faef3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faef4132af0 0x7faef412eef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52338/0 (socket says 172.21.15.116:52338) 2024-08-16T02:42:16.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.016+0000 7faef8cee640 1 -- 172.21.15.116:0/2800608195 learned_addr learned my addr 172.21.15.116:0/2800608195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef8cee640 1 -- 172.21.15.116:0/2800608195 >> v1:172.21.15.116:6789/0 conn(0x7faef4132af0 legacy=0x7faef412eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef8cee640 1 -- 172.21.15.116:0/2800608195 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef406d990 con 0x7faef41321a0 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef8cee640 1 --2- 172.21.15.116:0/2800608195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef41325b0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501968aca7e9e01f server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef37fe640 1 -- 172.21.15.116:0/2800608195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faee4009e40 con 0x7faef41321a0 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef37fe640 1 -- 172.21.15.116:0/2800608195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faee400e230 con 0x7faef41321a0 2024-08-16T02:42:16.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.017+0000 7faef37fe640 1 -- 172.21.15.116:0/2800608195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faee400e4d0 con 0x7faef41321a0 2024-08-16T02:42:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 -- 172.21.15.116:0/2800608195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 msgr2=0x7faef41325b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 --2- 172.21.15.116:0/2800608195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef41325b0 crc :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 -- 172.21.15.116:0/2800608195 shutdown_connections 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 --2- 172.21.15.116:0/2800608195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef41325b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 -- 172.21.15.116:0/2800608195 >> 172.21.15.116:0/2800608195 conn(0x7faef409f1d0 msgr2=0x7faef409f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 -- 172.21.15.116:0/2800608195 shutdown_connections 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 1 -- 172.21.15.116:0/2800608195 wait complete. 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.018+0000 7faefaf52640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.019+0000 7faefaf52640 1 Processor -- start 2024-08-16T02:42:16.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.019+0000 7faefaf52640 1 -- start start 2024-08-16T02:42:16.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faefaf52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faefaf52640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faef4120cd0 con 0x7faef41321a0 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef8cee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef8cee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56598/0 (socket says 172.21.15.116:56598) 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef8cee640 1 -- 172.21.15.116:0/2557536232 learned_addr learned my addr 172.21.15.116:0/2557536232 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef8cee640 1 -- 172.21.15.116:0/2557536232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef409dec0 con 0x7faef41321a0 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef8cee640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faee400edc0 con 0x7faef41321a0 2024-08-16T02:42:16.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7faee4015420 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef406f370 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7faee40156c0 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.021+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef4120540 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.022+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7faee4015870 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.022+0000 7faef17fa640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 0x7faee0040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.022+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7faee404fd20 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.022+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef41325b0 con 0x7faef41321a0 2024-08-16T02:42:16.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.023+0000 7faef3fff640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 0x7faee0040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.023+0000 7faef3fff640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 0x7faee0040220 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.028+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7faee4010a00 con 0x7faef41321a0 2024-08-16T02:42:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.214+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7faef4074670 con 0x7faee003dd10 2024-08-16T02:42:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.216+0000 7faef17fa640 1 -- 172.21.15.116:0/2557536232 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7faee8005870 con 0x7faee003dd10 2024-08-16T02:42:16.216 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 2 to deep-scrub 2024-08-16T02:42:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 msgr2=0x7faee0040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 0x7faee0040220 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 msgr2=0x7faef411dd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 crc :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 shutdown_connections 2024-08-16T02:42:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7faee003dd10 0x7faee0040220 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 --2- 172.21.15.116:0/2557536232 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faef41321a0 0x7faef411dd20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 >> 172.21.15.116:0/2557536232 conn(0x7faef409f1d0 msgr2=0x7faef41ba3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 shutdown_connections 2024-08-16T02:42:16.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.217+0000 7faefaf52640 1 -- 172.21.15.116:0/2557536232 wait complete. 2024-08-16T02:42:16.230 INFO:tasks.ceph:Scrubbing osd.3 2024-08-16T02:42:16.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-08-16T02:42:16.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.334+0000 7fec6ff92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.343+0000 7fec6ff92640 1 Processor -- start 2024-08-16T02:42:16.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.343+0000 7fec6ff92640 1 -- start start 2024-08-16T02:42:16.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6ff92640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec6809ba20 0x7fec6809be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6ff92640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec6809cfc0 con 0x7fec6809c370 2024-08-16T02:42:16.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6ff92640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fec6809d8a0 con 0x7fec6809ba20 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6dd2e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fec6809ba20 0x7fec6809be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6d52d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fec6809c370 0x7fec68133510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52354/0 (socket says 172.21.15.116:52354) 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6d52d640 1 -- 172.21.15.116:0/148964561 learned_addr learned my addr 172.21.15.116:0/148964561 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6dd2e640 1 -- 172.21.15.116:0/148964561 >> v1:172.21.15.116:6789/0 conn(0x7fec6809c370 legacy=0x7fec68133510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6cd2c640 1 -- 172.21.15.116:0/148964561 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1168307084 0 0) 0x7fec6809cfc0 con 0x7fec6809c370 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6dd2e640 1 -- 172.21.15.116:0/148964561 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec6809fbd0 con 0x7fec6809ba20 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6dd2e640 1 --2- 172.21.15.116:0/148964561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec6809be30 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ccfcdffaa866ed server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6cd2c640 1 -- 172.21.15.116:0/148964561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fec64009d90 con 0x7fec6809ba20 2024-08-16T02:42:16.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6cd2c640 1 -- 172.21.15.116:0/148964561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fec6400e230 con 0x7fec6809ba20 2024-08-16T02:42:16.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.345+0000 7fec6cd2c640 1 -- 172.21.15.116:0/148964561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fec6400e4d0 con 0x7fec6809ba20 2024-08-16T02:42:16.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 -- 172.21.15.116:0/148964561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 msgr2=0x7fec6809be30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 --2- 172.21.15.116:0/148964561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec6809be30 crc :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 -- 172.21.15.116:0/148964561 shutdown_connections 2024-08-16T02:42:16.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 --2- 172.21.15.116:0/148964561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec6809be30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 -- 172.21.15.116:0/148964561 >> 172.21.15.116:0/148964561 conn(0x7fec6806ede0 msgr2=0x7fec6806f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 -- 172.21.15.116:0/148964561 shutdown_connections 2024-08-16T02:42:16.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.346+0000 7fec6ff92640 1 -- 172.21.15.116:0/148964561 wait complete. 2024-08-16T02:42:16.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.347+0000 7fec6ff92640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.348+0000 7fec6ff92640 1 Processor -- start 2024-08-16T02:42:16.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.348+0000 7fec6ff92640 1 -- start start 2024-08-16T02:42:16.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6ff92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6ff92640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec68120e50 con 0x7fec6809ba20 2024-08-16T02:42:16.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6dd2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6dd2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56618/0 (socket says 172.21.15.116:56618) 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6dd2e640 1 -- 172.21.15.116:0/2860491320 learned_addr learned my addr 172.21.15.116:0/2860491320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6dd2e640 1 -- 172.21.15.116:0/2860491320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec681213e0 con 0x7fec6809ba20 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6dd2e640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fec64014620 con 0x7fec6809ba20 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fec64014780 con 0x7fec6809ba20 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fec64014a20 con 0x7fec6809ba20 2024-08-16T02:42:16.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.349+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec681228d0 con 0x7fec6809ba20 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.350+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec68123180 con 0x7fec6809ba20 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.350+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec6809d590 con 0x7fec6809ba20 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.350+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fec64014bd0 con 0x7fec6809ba20 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec5e7fc640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 0x7fec4c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fec64019040 con 0x7fec6809ba20 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec5e7fc640 1 --2- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 0x7fec4c0451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 --> v2:172.21.15.152:6804/2721550083 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec4c045850 con 0x7fec4c042d10 2024-08-16T02:42:16.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=228) ==== 24+0+0 (crc 0 0 0) 0x7fec64052420 con 0x7fec6809ba20 2024-08-16T02:42:16.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.351+0000 7fec6d52d640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 0x7fec4c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.352+0000 7fec6d52d640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 0x7fec4c040220 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.352+0000 7fec6e52f640 1 --2- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 0x7fec4c0451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-08-16T02:42:16.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.357+0000 7fec6e52f640 1 --2- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 0x7fec4c0451c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.358+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== osd.3 v2:172.21.15.152:6804/2721550083 1 ==== command_reply(tid 1: 0 ) ==== 8+0+5172 (crc 0 0 0) 0x7fec4c045850 con 0x7fec4c042d10 2024-08-16T02:42:16.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.362+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 --> v2:172.21.15.152:6804/2721550083 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fec6806cd80 con 0x7fec4c042d10 2024-08-16T02:42:16.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec5e7fc640 1 -- 172.21.15.116:0/2860491320 <== osd.3 v2:172.21.15.152:6804/2721550083 2 ==== command_reply(tid 2: 0 ) ==== 8+0+22 (crc 0 0 0) 0x7fec6000a820 con 0x7fec4c042d10 2024-08-16T02:42:16.365 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-08-16T02:42:16.365 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "" 2024-08-16T02:42:16.365 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-08-16T02:42:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 msgr2=0x7fec4c0451c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 0x7fec4c0451c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 msgr2=0x7fec4c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 0x7fec4c040220 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 msgr2=0x7fec681217a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 crc :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 shutdown_connections 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> v2:172.21.15.152:6804/2721550083 conn(0x7fec4c042d10 0x7fec4c0451c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fec4c03dd10 0x7fec4c040220 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 --2- 172.21.15.116:0/2860491320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec6809ba20 0x7fec681217a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 >> 172.21.15.116:0/2860491320 conn(0x7fec6806ede0 msgr2=0x7fec6811a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 shutdown_connections 2024-08-16T02:42:16.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.364+0000 7fec6ff92640 1 -- 172.21.15.116:0/2860491320 wait complete. 2024-08-16T02:42:16.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-08-16T02:42:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.479+0000 7f3e0c792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.484+0000 7f3e0c792640 1 Processor -- start 2024-08-16T02:42:16.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.484+0000 7f3e0c792640 1 -- start start 2024-08-16T02:42:16.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0c792640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e041384c0 0x7f3e0413a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0c792640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0406fba0 con 0x7f3e04135570 2024-08-16T02:42:16.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0c792640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3e040a2f30 con 0x7f3e041384c0 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e041384c0 0x7f3e0413a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e041384c0 0x7f3e0413a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56630/0 (socket says 172.21.15.116:56630) 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0a52e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3e04135570 0x7f3e04135980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52364/0 (socket says 172.21.15.116:52364) 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 -- 172.21.15.116:0/1401777301 learned_addr learned my addr 172.21.15.116:0/1401777301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 -- 172.21.15.116:0/1401777301 >> v1:172.21.15.116:6789/0 conn(0x7f3e04135570 legacy=0x7f3e04135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 -- 172.21.15.116:0/1401777301 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e040a3920 con 0x7f3e041384c0 2024-08-16T02:42:16.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0952c640 1 -- 172.21.15.116:0/1401777301 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420294168 0 0) 0x7f3e0406fba0 con 0x7f3e04135570 2024-08-16T02:42:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e09d2d640 1 --2- 172.21.15.116:0/1401777301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e041384c0 0x7f3e0413a8e0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e536a1d8b6a04f55 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0952c640 1 -- 172.21.15.116:0/1401777301 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3df4009e50 con 0x7f3e041384c0 2024-08-16T02:42:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0952c640 1 -- 172.21.15.116:0/1401777301 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3df400e230 con 0x7f3e041384c0 2024-08-16T02:42:16.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.486+0000 7f3e0952c640 1 -- 172.21.15.116:0/1401777301 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3df400e4d0 con 0x7f3e041384c0 2024-08-16T02:42:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 -- 172.21.15.116:0/1401777301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e041384c0 msgr2=0x7f3e0413a8e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1401777301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e041384c0 0x7f3e0413a8e0 crc :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 -- 172.21.15.116:0/1401777301 shutdown_connections 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1401777301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e041384c0 0x7f3e0413a8e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 -- 172.21.15.116:0/1401777301 >> 172.21.15.116:0/1401777301 conn(0x7f3e040a5960 msgr2=0x7f3e040a5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 -- 172.21.15.116:0/1401777301 shutdown_connections 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.487+0000 7f3e0c792640 1 -- 172.21.15.116:0/1401777301 wait complete. 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.488+0000 7f3e0c792640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.489+0000 7f3e0c792640 1 Processor -- start 2024-08-16T02:42:16.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.489+0000 7f3e0c792640 1 -- start start 2024-08-16T02:42:16.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0c792640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0c792640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e041350b0 con 0x7f3e04135570 2024-08-16T02:42:16.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0a52e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0a52e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56638/0 (socket says 172.21.15.116:56638) 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0a52e640 1 -- 172.21.15.116:0/1313692746 learned_addr learned my addr 172.21.15.116:0/1313692746 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0a52e640 1 -- 172.21.15.116:0/1313692746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e04125ad0 con 0x7f3e04135570 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.490+0000 7f3e0a52e640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.491+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e00004b80 con 0x7f3e04135570 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.491+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f3e00004ce0 con 0x7f3e04135570 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.491+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f3e00007120 con 0x7f3e04135570 2024-08-16T02:42:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.491+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e04122160 con 0x7f3e04135570 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.491+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e04124400 con 0x7f3e04135570 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.492+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f3e000072d0 con 0x7f3e04135570 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.492+0000 7f3dfaffd640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 0x7f3de8040250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.492+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e04123dc0 con 0x7f3e04135570 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.492+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f3e0004e860 con 0x7f3e04135570 2024-08-16T02:42:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.492+0000 7f3e09d2d640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 0x7f3de8040250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.493+0000 7f3e09d2d640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 0x7f3de8040250 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.497+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f3e0000adb0 con 0x7f3e04135570 2024-08-16T02:42:16.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.681+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3e0406f420 con 0x7f3de803dd40 2024-08-16T02:42:16.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.682+0000 7f3dfaffd640 1 -- 172.21.15.116:0/1313692746 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (crc 0 0 0) 0x7f3df4015420 con 0x7f3de803dd40 2024-08-16T02:42:16.683 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 3 to deep-scrub 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.684+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 msgr2=0x7f3de8040250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.684+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 0x7f3de8040250 crc :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.684+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 msgr2=0x7f3e04122690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.684+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 crc :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.684+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 shutdown_connections 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.685+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f3de803dd40 0x7f3de8040250 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.685+0000 7f3e0c792640 1 --2- 172.21.15.116:0/1313692746 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e04135570 0x7f3e04122690 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.685+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 >> 172.21.15.116:0/1313692746 conn(0x7f3e040a5960 msgr2=0x7f3e041362a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.685+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 shutdown_connections 2024-08-16T02:42:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.685+0000 7f3e0c792640 1 -- 172.21.15.116:0/1313692746 wait complete. 2024-08-16T02:42:16.697 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:42:16.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.801+0000 7fd291949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.806+0000 7fd291949640 1 Processor -- start 2024-08-16T02:42:16.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.806+0000 7fd291949640 1 -- start start 2024-08-16T02:42:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd291949640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd28c138490 0x7fd28c13a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd291949640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd28c0a4f10 con 0x7fd28c135490 2024-08-16T02:42:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd291949640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd28c135fb0 con 0x7fd28c138490 2024-08-16T02:42:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd2827fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd28c138490 0x7fd28c13a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd2827fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd28c138490 0x7fd28c13a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56654/0 (socket says 172.21.15.116:56654) 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd28affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd28c135490 0x7fd28c1358a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52374/0 (socket says 172.21.15.116:52374) 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.807+0000 7fd2827fc640 1 -- 172.21.15.116:0/2192035072 learned_addr learned my addr 172.21.15.116:0/2192035072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd2827fc640 1 -- 172.21.15.116:0/2192035072 >> v1:172.21.15.116:6789/0 conn(0x7fd28c135490 legacy=0x7fd28c1358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd2827fc640 1 -- 172.21.15.116:0/2192035072 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c072b20 con 0x7fd28c138490 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd28a7fc640 1 -- 172.21.15.116:0/2192035072 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405038765 0 0) 0x7fd28c0a4f10 con 0x7fd28c135490 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd2827fc640 1 --2- 172.21.15.116:0/2192035072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c138490 0x7fd28c13a8b0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efae02d653c45d7e server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd28a7fc640 1 -- 172.21.15.116:0/2192035072 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd274009e50 con 0x7fd28c138490 2024-08-16T02:42:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd28a7fc640 1 -- 172.21.15.116:0/2192035072 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd27400e230 con 0x7fd28c138490 2024-08-16T02:42:16.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.808+0000 7fd28a7fc640 1 -- 172.21.15.116:0/2192035072 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd27400e4d0 con 0x7fd28c138490 2024-08-16T02:42:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 -- 172.21.15.116:0/2192035072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c138490 msgr2=0x7fd28c13a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 --2- 172.21.15.116:0/2192035072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c138490 0x7fd28c13a8b0 crc :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 -- 172.21.15.116:0/2192035072 shutdown_connections 2024-08-16T02:42:16.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 --2- 172.21.15.116:0/2192035072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c138490 0x7fd28c13a8b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 -- 172.21.15.116:0/2192035072 >> 172.21.15.116:0/2192035072 conn(0x7fd28c0a5880 msgr2=0x7fd28c0a5c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 -- 172.21.15.116:0/2192035072 shutdown_connections 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 1 -- 172.21.15.116:0/2192035072 wait complete. 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.809+0000 7fd291949640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.810+0000 7fd291949640 1 Processor -- start 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.811+0000 7fd291949640 1 -- start start 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd291949640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd291949640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c120c50 con 0x7fd28c135490 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd28affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd28affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56668/0 (socket says 172.21.15.116:56668) 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd28affd640 1 -- 172.21.15.116:0/660868610 learned_addr learned my addr 172.21.15.116:0/660868610 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd28affd640 1 -- 172.21.15.116:0/660868610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c123000 con 0x7fd28c135490 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd28affd640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd27c004b80 con 0x7fd28c135490 2024-08-16T02:42:16.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fd27c004ce0 con 0x7fd28c135490 2024-08-16T02:42:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c122650 con 0x7fd28c135490 2024-08-16T02:42:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fd27c007120 con 0x7fd28c135490 2024-08-16T02:42:16.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.812+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c123ff0 con 0x7fd28c135490 2024-08-16T02:42:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.813+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fd27c0072d0 con 0x7fd28c135490 2024-08-16T02:42:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.813+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c1358a0 con 0x7fd28c135490 2024-08-16T02:42:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.813+0000 7fd288ff9640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 0x7fd26c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:16.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.814+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fd27c04e8e0 con 0x7fd28c135490 2024-08-16T02:42:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.814+0000 7fd2827fc640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 0x7fd26c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:16.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.815+0000 7fd2827fc640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 0x7fd26c040220 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:16.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:16.819+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fd27c00adb0 con 0x7fd28c135490 2024-08-16T02:42:17.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.002+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd28c120fe0 con 0x7fd26c03dd10 2024-08-16T02:42:17.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.003+0000 7fd288ff9640 1 -- 172.21.15.116:0/660868610 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31238 (crc 0 0 0) 0x7fd28c120fe0 con 0x7fd26c03dd10 2024-08-16T02:42:17.005 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:17.005 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:42:17.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 msgr2=0x7fd26c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:17.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 0x7fd26c040220 crc :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:17.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 msgr2=0x7fd28c1c9d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:17.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 crc :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 shutdown_connections 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fd26c03dd10 0x7fd26c040220 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 --2- 172.21.15.116:0/660868610 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd28c135490 0x7fd28c1c9d50 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 >> 172.21.15.116:0/660868610 conn(0x7fd28c0a5880 msgr2=0x7fd28c070f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 shutdown_connections 2024-08-16T02:42:17.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:17.006+0000 7fd291949640 1 -- 172.21.15.116:0/660868610 wait complete. 2024-08-16T02:42:17.018 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-08-16T02:42:15.729804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12880,"kb_used_data":12880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209178032,"statfs":{"total":214211493888,"available":214198304768,"internally_reserved":0,"allocated":13189120,"data_stored":13189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.437705"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.384710+0000","last_change":"2024-08-16T02:37:10.780912+0000","last_active":"2024-08-16T02:42:11.384710+0000","last_peered":"2024-08-16T02:42:11.384710+0000","last_clean":"2024-08-16T02:42:11.384710+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:42:11.384710+0000","last_undegraded":"2024-08-16T02:42:11.384710+0000","last_fullsized":"2024-08-16T02:42:11.384710+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.372334+0000","last_change":"2024-08-16T02:37:10.780752+0000","last_active":"2024-08-16T02:42:11.372334+0000","last_peered":"2024-08-16T02:42:11.372334+0000","last_clean":"2024-08-16T02:42:11.372334+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:42:11.372334+0000","last_undegraded":"2024-08-16T02:42:11.372334+0000","last_fullsized":"2024-08-16T02:42:11.372334+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.371593+0000","last_change":"2024-08-16T02:37:10.777832+0000","last_active":"2024-08-16T02:42:11.371593+0000","last_peered":"2024-08-16T02:42:11.371593+0000","last_clean":"2024-08-16T02:42:11.371593+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:42:11.371593+0000","last_undegraded":"2024-08-16T02:42:11.371593+0000","last_fullsized":"2024-08-16T02:42:11.371593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.330063+0000","last_change":"2024-08-16T02:37:10.776875+0000","last_active":"2024-08-16T02:42:11.330063+0000","last_peered":"2024-08-16T02:42:11.330063+0000","last_clean":"2024-08-16T02:42:11.330063+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:42:11.330063+0000","last_undegraded":"2024-08-16T02:42:11.330063+0000","last_fullsized":"2024-08-16T02:42:11.330063+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.375626+0000","last_change":"2024-08-16T02:37:10.776780+0000","last_active":"2024-08-16T02:42:11.375626+0000","last_peered":"2024-08-16T02:42:11.375626+0000","last_clean":"2024-08-16T02:42:11.375626+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:42:11.375626+0000","last_undegraded":"2024-08-16T02:42:11.375626+0000","last_fullsized":"2024-08-16T02:42:11.375626+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.370729+0000","last_change":"2024-08-16T02:37:10.775868+0000","last_active":"2024-08-16T02:42:11.370729+0000","last_peered":"2024-08-16T02:42:11.370729+0000","last_clean":"2024-08-16T02:42:11.370729+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:42:11.370729+0000","last_undegraded":"2024-08-16T02:42:11.370729+0000","last_fullsized":"2024-08-16T02:42:11.370729+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":654,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.380150+0000","last_change":"2024-08-16T02:37:10.776920+0000","last_active":"2024-08-16T02:42:11.380150+0000","last_peered":"2024-08-16T02:42:11.380150+0000","last_clean":"2024-08-16T02:42:11.380150+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:42:11.380150+0000","last_undegraded":"2024-08-16T02:42:11.380150+0000","last_fullsized":"2024-08-16T02:42:11.380150+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":655,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.398074+0000","last_change":"2024-08-16T02:37:10.780792+0000","last_active":"2024-08-16T02:42:11.398074+0000","last_peered":"2024-08-16T02:42:11.398074+0000","last_clean":"2024-08-16T02:42:11.398074+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:42:11.398074+0000","last_undegraded":"2024-08-16T02:42:11.398074+0000","last_fullsized":"2024-08-16T02:42:11.398074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:07.728483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":726,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:11.344434+0000","last_change":"2024-08-16T02:37:06.794309+0000","last_active":"2024-08-16T02:42:11.344434+0000","last_peered":"2024-08-16T02:42:11.344434+0000","last_clean":"2024-08-16T02:42:11.344434+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:42:11.344434+0000","last_undegraded":"2024-08-16T02:42:11.344434+0000","last_fullsized":"2024-08-16T02:42:11.344434+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:37:05.720602+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198304768,"internally_reserved":0,"allocated":13189120,"data_stored":13189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3472,"kb_used_data":3472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294256,"statfs":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2832,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294896,"statfs":{"total":53552873472,"available":53549973504,"internally_reserved":0,"allocated":2899968,"data_stored":2899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2828,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294900,"statfs":{"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3748,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293980,"statfs":{"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549973504,"internally_reserved":0,"allocated":2899968,"data_stored":2899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:42:17.020 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.021 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.021 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.021 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.021 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.021 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.022 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.022 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-08-16T02:37:07.728483+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=7, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.022 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-08-16T02:37:05.720602+0000 time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=37, tm_sec=5, tm_wday=4, tm_yday=229, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=8, tm_mday=16, tm_hour=2, tm_min=42, tm_sec=13, tm_wday=4, tm_yday=229, tm_isdst=0) 2024-08-16T02:42:17.022 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-08-16T02:42:37.024 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-16T02:42:37.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.150+0000 7fc3c24ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.155+0000 7fc3c24ed640 1 Processor -- start 2024-08-16T02:42:37.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.156+0000 7fc3c24ed640 1 -- start start 2024-08-16T02:42:37.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3c24ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3c24ed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3bc06cf10 con 0x7fc3bc135600 2024-08-16T02:42:37.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3c24ed640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc3bc136120 con 0x7fc3bc1384a0 2024-08-16T02:42:37.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59032/0 (socket says 172.21.15.116:59032) 2024-08-16T02:42:37.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 -- 172.21.15.116:0/1163482785 learned_addr learned my addr 172.21.15.116:0/1163482785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 -- 172.21.15.116:0/1163482785 >> v1:172.21.15.116:6789/0 conn(0x7fc3bc135600 legacy=0x7fc3bc135a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 -- 172.21.15.116:0/1163482785 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc1351d0 con 0x7fc3bc1384a0 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3bb7fe640 1 -- 172.21.15.116:0/1163482785 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49279669 0 0) 0x7fc3bc06cf10 con 0x7fc3bc135600 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3b3fff640 1 --2- 172.21.15.116:0/1163482785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e515ffda7d73c8 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3bb7fe640 1 -- 172.21.15.116:0/1163482785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc3a4009e50 con 0x7fc3bc1384a0 2024-08-16T02:42:37.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3bb7fe640 1 -- 172.21.15.116:0/1163482785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc3a400e230 con 0x7fc3bc1384a0 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.157+0000 7fc3bb7fe640 1 -- 172.21.15.116:0/1163482785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc3a400e4d0 con 0x7fc3bc1384a0 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 -- 172.21.15.116:0/1163482785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc1384a0 msgr2=0x7fc3bc13a8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/1163482785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 crc :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 -- 172.21.15.116:0/1163482785 shutdown_connections 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/1163482785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc1384a0 0x7fc3bc13a8c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 -- 172.21.15.116:0/1163482785 >> 172.21.15.116:0/1163482785 conn(0x7fc3bc06ec10 msgr2=0x7fc3bc06f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:37.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 -- 172.21.15.116:0/1163482785 shutdown_connections 2024-08-16T02:42:37.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.158+0000 7fc3c24ed640 1 -- 172.21.15.116:0/1163482785 wait complete. 2024-08-16T02:42:37.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.159+0000 7fc3c24ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.160+0000 7fc3c24ed640 1 Processor -- start 2024-08-16T02:42:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.160+0000 7fc3c24ed640 1 -- start start 2024-08-16T02:42:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3c24ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3c24ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc3bc120d80 con 0x7fc3bc135600 2024-08-16T02:42:37.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59038/0 (socket says 172.21.15.116:59038) 2024-08-16T02:42:37.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3bbfff640 1 -- 172.21.15.116:0/2028219744 learned_addr learned my addr 172.21.15.116:0/2028219744 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:37.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.161+0000 7fc3bbfff640 1 -- 172.21.15.116:0/2028219744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3bc121ba0 con 0x7fc3bc135600 2024-08-16T02:42:37.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3bbfff640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc3ac0051e0 con 0x7fc3bc135600 2024-08-16T02:42:37.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7fc3ac005340 con 0x7fc3bc135600 2024-08-16T02:42:37.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7fc3ac00c6d0 con 0x7fc3bc135600 2024-08-16T02:42:37.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3bc11f340 con 0x7fc3bc135600 2024-08-16T02:42:37.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.162+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3bc122700 con 0x7fc3bc135600 2024-08-16T02:42:37.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.163+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7fc3ac00c880 con 0x7fc3bc135600 2024-08-16T02:42:37.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.163+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3bc06a020 con 0x7fc3bc135600 2024-08-16T02:42:37.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.163+0000 7fc3b97fa640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 0x7fc39c040220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.163+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7fc3ac04e650 con 0x7fc3bc135600 2024-08-16T02:42:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.164+0000 7fc3b3fff640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 0x7fc39c040220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.164+0000 7fc3b3fff640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 0x7fc39c040220 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.168+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7fc3ac00da20 con 0x7fc3bc135600 2024-08-16T02:42:37.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.351+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 --> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3bc09d420 con 0x7fc39c03dd10 2024-08-16T02:42:37.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.352+0000 7fc3b97fa640 1 -- 172.21.15.116:0/2028219744 <== mgr.4100 v2:172.21.15.152:6800/3181043282 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+31243 (crc 0 0 0) 0x7fc3a4009e70 con 0x7fc39c03dd10 2024-08-16T02:42:37.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-08-16T02:42:37.354 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-08-16T02:42:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 msgr2=0x7fc39c040220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 0x7fc39c040220 crc :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 msgr2=0x7fc3bc11fc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 crc :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 shutdown_connections 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7fc39c03dd10 0x7fc39c040220 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 --2- 172.21.15.116:0/2028219744 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3bc135600 0x7fc3bc11fc80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 >> 172.21.15.116:0/2028219744 conn(0x7fc3bc06ec10 msgr2=0x7fc3bc133e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 shutdown_connections 2024-08-16T02:42:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.354+0000 7fc3c24ed640 1 -- 172.21.15.116:0/2028219744 wait complete. 2024-08-16T02:42:37.366 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-08-16T02:42:35.734627+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209190912,"kb_used":12880,"kb_used_data":12880,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":209178032,"statfs":{"total":214211493888,"available":214198304768,"internally_reserved":0,"allocated":13189120,"data_stored":13189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002769"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:15.740705+0000","last_change":"2024-08-16T02:42:15.768653+0000","last_active":"2024-08-16T02:42:15.740705+0000","last_peered":"2024-08-16T02:42:15.740705+0000","last_clean":"2024-08-16T02:42:15.740705+0000","last_became_active":"2024-08-16T02:37:08.828101+0000","last_became_peered":"2024-08-16T02:37:08.828101+0000","last_unstale":"2024-08-16T02:42:15.740705+0000","last_undegraded":"2024-08-16T02:42:15.740705+0000","last_fullsized":"2024-08-16T02:42:15.740705+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:15.768483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:15.768483+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:16.221055+0000","last_change":"2024-08-16T02:42:16.232987+0000","last_active":"2024-08-16T02:42:16.221055+0000","last_peered":"2024-08-16T02:42:16.221055+0000","last_clean":"2024-08-16T02:42:16.221055+0000","last_became_active":"2024-08-16T02:37:08.831977+0000","last_became_peered":"2024-08-16T02:37:08.831977+0000","last_unstale":"2024-08-16T02:42:16.221055+0000","last_undegraded":"2024-08-16T02:42:16.221055+0000","last_fullsized":"2024-08-16T02:42:16.221055+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:16.232741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:16.232741+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:16.688198+0000","last_change":"2024-08-16T02:42:16.702824+0000","last_active":"2024-08-16T02:42:16.688198+0000","last_peered":"2024-08-16T02:42:16.688198+0000","last_clean":"2024-08-16T02:42:16.688198+0000","last_became_active":"2024-08-16T02:37:08.824719+0000","last_became_peered":"2024-08-16T02:37:08.824719+0000","last_unstale":"2024-08-16T02:42:16.688198+0000","last_undegraded":"2024-08-16T02:42:16.688198+0000","last_fullsized":"2024-08-16T02:42:16.688198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:16.702613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:16.702613+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:16.218379+0000","last_change":"2024-08-16T02:42:16.234029+0000","last_active":"2024-08-16T02:42:16.218379+0000","last_peered":"2024-08-16T02:42:16.218379+0000","last_clean":"2024-08-16T02:42:16.218379+0000","last_became_active":"2024-08-16T02:37:08.809215+0000","last_became_peered":"2024-08-16T02:37:08.809215+0000","last_unstale":"2024-08-16T02:42:16.218379+0000","last_undegraded":"2024-08-16T02:42:16.218379+0000","last_fullsized":"2024-08-16T02:42:16.218379+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:16.233820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:16.233820+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":658,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:15.267688+0000","last_change":"2024-08-16T02:42:15.286624+0000","last_active":"2024-08-16T02:42:15.267688+0000","last_peered":"2024-08-16T02:42:15.267688+0000","last_clean":"2024-08-16T02:42:15.267688+0000","last_became_active":"2024-08-16T02:37:08.825744+0000","last_became_peered":"2024-08-16T02:37:08.825744+0000","last_unstale":"2024-08-16T02:42:15.267688+0000","last_undegraded":"2024-08-16T02:42:15.267688+0000","last_fullsized":"2024-08-16T02:42:15.267688+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"15'2","last_scrub_stamp":"2024-08-16T02:42:15.286431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:15.286431+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:15.738644+0000","last_change":"2024-08-16T02:42:15.766691+0000","last_active":"2024-08-16T02:42:15.738644+0000","last_peered":"2024-08-16T02:42:15.738644+0000","last_clean":"2024-08-16T02:42:15.738644+0000","last_became_active":"2024-08-16T02:37:08.810211+0000","last_became_peered":"2024-08-16T02:37:08.810211+0000","last_unstale":"2024-08-16T02:42:15.738644+0000","last_undegraded":"2024-08-16T02:42:15.738644+0000","last_fullsized":"2024-08-16T02:42:15.738644+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:15.766529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:15.766529+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":656,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:15.267822+0000","last_change":"2024-08-16T02:42:15.278865+0000","last_active":"2024-08-16T02:42:15.267822+0000","last_peered":"2024-08-16T02:42:15.267822+0000","last_clean":"2024-08-16T02:42:15.267822+0000","last_became_active":"2024-08-16T02:37:08.810265+0000","last_became_peered":"2024-08-16T02:37:08.810265+0000","last_unstale":"2024-08-16T02:42:15.267822+0000","last_undegraded":"2024-08-16T02:42:15.267822+0000","last_fullsized":"2024-08-16T02:42:15.267822+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-16T02:42:15.278697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:15.278697+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":657,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:15.740532+0000","last_change":"2024-08-16T02:42:15.769382+0000","last_active":"2024-08-16T02:42:15.740532+0000","last_peered":"2024-08-16T02:42:15.740532+0000","last_clean":"2024-08-16T02:42:15.740532+0000","last_became_active":"2024-08-16T02:37:08.815743+0000","last_became_peered":"2024-08-16T02:37:08.815743+0000","last_unstale":"2024-08-16T02:42:15.740532+0000","last_undegraded":"2024-08-16T02:42:15.740532+0000","last_fullsized":"2024-08-16T02:42:15.740532+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"13'1","last_scrub_stamp":"2024-08-16T02:42:15.769213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:15.769213+0000","last_clean_scrub_stamp":"2024-08-16T02:37:07.728483+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":728,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-08-16T02:42:16.685764+0000","last_change":"2024-08-16T02:42:16.716079+0000","last_active":"2024-08-16T02:42:16.685764+0000","last_peered":"2024-08-16T02:42:16.685764+0000","last_clean":"2024-08-16T02:42:16.685764+0000","last_became_active":"2024-08-16T02:37:06.775953+0000","last_became_peered":"2024-08-16T02:37:06.775953+0000","last_unstale":"2024-08-16T02:42:16.685764+0000","last_undegraded":"2024-08-16T02:42:16.685764+0000","last_fullsized":"2024-08-16T02:42:16.685764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"12'52","last_scrub_stamp":"2024-08-16T02:42:16.715846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-16T02:42:16.715846+0000","last_clean_scrub_stamp":"2024-08-16T02:37:05.720602+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214211493888,"available":214198304768,"internally_reserved":0,"allocated":13189120,"data_stored":13189120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3472,"kb_used_data":3472,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294256,"statfs":{"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2832,"kb_used_data":2832,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294896,"statfs":{"total":53552873472,"available":53549973504,"internally_reserved":0,"allocated":2899968,"data_stored":2899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":2828,"kb_used_data":2828,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52294900,"statfs":{"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705993,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":3748,"kb_used_data":3748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52293980,"statfs":{"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53549035520,"internally_reserved":0,"allocated":3837952,"data_stored":3837952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53549977600,"internally_reserved":0,"allocated":2895872,"data_stored":2895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53549973504,"internally_reserved":0,"allocated":2899968,"data_stored":2899968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53552873472,"available":53549318144,"internally_reserved":0,"allocated":3555328,"data_stored":3555328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-16T02:42:37.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-08-16T02:42:37.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.469+0000 7f42e4cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:37.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.477+0000 7f42e4cad640 1 Processor -- start 2024-08-16T02:42:37.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.477+0000 7f42e4cad640 1 -- start start 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42e4cad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42e01302c0 0x7f42e01306d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42e4cad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e00708b0 con 0x7f42e009dc30 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42e4cad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f42e006f020 con 0x7f42e01302c0 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42de59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42e01302c0 0x7f42e01306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42de59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42e01302c0 0x7f42e01306d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59040/0 (socket says 172.21.15.116:59040) 2024-08-16T02:42:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42d5d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f42e009dc30 0x7f42e009e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34810/0 (socket says 172.21.15.116:34810) 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42de59c640 1 -- 172.21.15.116:0/3002985082 learned_addr learned my addr 172.21.15.116:0/3002985082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42de59c640 1 -- 172.21.15.116:0/3002985082 >> v1:172.21.15.116:6789/0 conn(0x7f42e009dc30 legacy=0x7f42e009e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.478+0000 7f42de59c640 1 -- 172.21.15.116:0/3002985082 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e006f200 con 0x7f42e01302c0 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42de59c640 1 --2- 172.21.15.116:0/3002985082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e01302c0 0x7f42e01306d0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3580bcce5811c01 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42ddd9b640 1 -- 172.21.15.116:0/3002985082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f42d0009e50 con 0x7f42e01302c0 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42ddd9b640 1 -- 172.21.15.116:0/3002985082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f42d000e230 con 0x7f42e01302c0 2024-08-16T02:42:37.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42ddd9b640 1 -- 172.21.15.116:0/3002985082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f42d000e4d0 con 0x7f42e01302c0 2024-08-16T02:42:37.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42e4cad640 1 -- 172.21.15.116:0/3002985082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e01302c0 msgr2=0x7f42e01306d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.479+0000 7f42e4cad640 1 --2- 172.21.15.116:0/3002985082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e01302c0 0x7f42e01306d0 crc :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 1 -- 172.21.15.116:0/3002985082 shutdown_connections 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 1 --2- 172.21.15.116:0/3002985082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e01302c0 0x7f42e01306d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 1 -- 172.21.15.116:0/3002985082 >> 172.21.15.116:0/3002985082 conn(0x7f42e009f1d0 msgr2=0x7f42e009f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 1 -- 172.21.15.116:0/3002985082 shutdown_connections 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 1 -- 172.21.15.116:0/3002985082 wait complete. 2024-08-16T02:42:37.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.480+0000 7f42e4cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-08-16T02:42:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.481+0000 7f42e4cad640 1 Processor -- start 2024-08-16T02:42:37.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.481+0000 7f42e4cad640 1 -- start start 2024-08-16T02:42:37.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.482+0000 7f42e4cad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.482+0000 7f42e4cad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42e0122f80 con 0x7f42e009dc30 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59056/0 (socket says 172.21.15.116:59056) 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42de59c640 1 -- 172.21.15.116:0/734016619 learned_addr learned my addr 172.21.15.116:0/734016619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42de59c640 1 -- 172.21.15.116:0/734016619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e0073040 con 0x7f42e009dc30 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42de59c640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f42d000eda0 con 0x7f42e009dc30 2024-08-16T02:42:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (crc 0 0 0) 0x7f42d0015420 con 0x7f42e009dc30 2024-08-16T02:42:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e0120990 con 0x7f42e009dc30 2024-08-16T02:42:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (crc 0 0 0) 0x7f42d00156c0 con 0x7f42e009dc30 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.483+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e0121da0 con 0x7f42e009dc30 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.484+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e0121270 con 0x7f42e009dc30 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.484+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 49880+0+0 (crc 0 0 0) 0x7f42d00158c0 con 0x7f42e009dc30 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.485+0000 7f42d77fe640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 0x7f42c0040200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.485+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(228..228 src has 178..228) ==== 2700+0+0 (crc 0 0 0) 0x7f42d004fc40 con 0x7f42e009dc30 2024-08-16T02:42:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.485+0000 7f42d5d9b640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 0x7f42c0040200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-16T02:42:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.486+0000 7f42d5d9b640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 0x7f42c0040200 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-16T02:42:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.490+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+194988 (crc 0 0 0) 0x7f42d0010a00 con 0x7f42e009dc30 2024-08-16T02:42:37.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.675+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f42e0073040 con 0x7f42e009dc30 2024-08-16T02:42:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.679+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) ==== 123+0+0 (crc 0 0 0) 0x7f42e0073040 con 0x7f42e009dc30 2024-08-16T02:42:37.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.680+0000 7f42d77fe640 1 -- 172.21.15.116:0/734016619 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(1 keys) ==== 35+0+0 (crc 0 0 0) 0x7f42d00839c0 con 0x7f42e009dc30 2024-08-16T02:42:37.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.682+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 msgr2=0x7f42c0040200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.682+0000 7f42e4cad640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 0x7f42c0040200 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.682+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 msgr2=0x7f42e011f040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.682+0000 7f42e4cad640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 crc :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 shutdown_connections 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.152:6800/3181043282,v1:172.21.15.152:6801/3181043282] conn(0x7f42c003dcf0 0x7f42c0040200 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 --2- 172.21.15.116:0/734016619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42e009dc30 0x7f42e011f040 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 >> 172.21.15.116:0/734016619 conn(0x7f42e009f1d0 msgr2=0x7f42e0072400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-16T02:42:37.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 shutdown_connections 2024-08-16T02:42:37.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-08-16T02:42:37.683+0000 7f42e4cad640 1 -- 172.21.15.116:0/734016619 wait complete. 2024-08-16T02:42:37.698 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-16T02:42:37.698 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-16T02:42:37.698 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-16T02:42:37.699 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:37.774 INFO:tasks.ceph.osd.0:Stopped 2024-08-16T02:42:37.774 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-16T02:42:37.774 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:37.852 INFO:tasks.ceph.osd.1:Stopped 2024-08-16T02:42:37.852 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-16T02:42:37.852 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:37.923 INFO:tasks.ceph.osd.2:Stopped 2024-08-16T02:42:37.924 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-16T02:42:37.924 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:38.008 INFO:tasks.ceph.osd.3:Stopped 2024-08-16T02:42:38.008 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-16T02:42:38.009 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-16T02:42:38.009 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:38.040 INFO:tasks.ceph.mgr.x:Stopped 2024-08-16T02:42:38.040 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-16T02:42:38.041 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-16T02:42:38.041 INFO:teuthology.orchestra.run:waiting for 300 2024-08-16T02:42:38.053 INFO:tasks.ceph.mon.a:Stopped 2024-08-16T02:42:38.053 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-16T02:42:38.054 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-08-16T02:42:38.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:42:38.121 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-16T02:42:38.333 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:42:38.333 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-16T02:42:38.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:42:38.423 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-16T02:42:38.575 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:42:38.575 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-16T02:42:38.639 INFO:tasks.ceph:Archiving mon data... 2024-08-16T02:42:38.640 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/data/mon.a.tgz 2024-08-16T02:42:38.641 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-08-16T02:42:38.659 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.ynMRPdDbWD 2024-08-16T02:42:38.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ynMRPdDbWD 2024-08-16T02:42:38.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.ynMRPdDbWD 2024-08-16T02:42:38.913 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.ynMRPdDbWD is 357KB 2024-08-16T02:42:39.010 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.ynMRPdDbWD 2024-08-16T02:42:39.025 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-16T02:42:39.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:42:39.067 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-16T02:42:39.145 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:42:39.161 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-16T02:42:39.161 INFO:tasks.ceph:Archiving crash dumps... 2024-08-16T02:42:39.162 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi116/crash 2024-08-16T02:42:39.163 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:42:39.209 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi152/crash 2024-08-16T02:42:39.210 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-16T02:42:39.243 INFO:tasks.ceph:Compressing logs... 2024-08-16T02:42:39.244 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:42:39.254 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-16T02:42:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-08-16T02:42:39.276 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32826.log 2024-08-16T02:42:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32448.log 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30582.log 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr: 86.0%/var/log/ceph/ceph-client.admin.32448.log: -- replaced with /var/log/ceph/ceph-client.admin.32826.log.gz 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.32448.log.gz 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30581.log 2024-08-16T02:42:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.30582.log.gz 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30438.log 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32658.log 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30900.log 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30581.log.gz 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30438.log.gz 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30749.log 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-16T02:42:39.292 INFO:teuthology.orchestra.run.smithi116.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.32658.log.gz 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30900.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.30900.log.gz 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30579.log 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30749.log.gz 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29972.log 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30579.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29106.log 2024-08-16T02:42:39.293 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.29972.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30579.log.gz 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30214.log 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.29972.log.gz 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.tmp-client.admin.29106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33034.log 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29106.log.gz 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30214.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33034.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.30214.log.gz 2024-08-16T02:42:39.294 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.33034.log.gz 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30303.log 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30942.log 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30480.log 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30942.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.30303.log.gz 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32742.log 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30942.log.gz 2024-08-16T02:42:39.295 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30750.log 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30480.log: /var/log/ceph/ceph-client.admin.32742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30270.log 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30480.log.gz 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr: 85.8%/var/log/ceph/ceph-client.admin.30750.log: -- replaced with /var/log/ceph/ceph-client.admin.32742.log.gz 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.30750.log.gz 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32616.log 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30270.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.30270.log.gz 2024-08-16T02:42:39.296 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30157.log 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32616.log.gz 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32532.log 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30157.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32700.log 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.30157.log.gz 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32994.log 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30747.log 2024-08-16T02:42:39.297 INFO:teuthology.orchestra.run.smithi116.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.32532.log.gz 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32700.log: /var/log/ceph/ceph-client.admin.32994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31162.log 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.32700.log.gz 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.32994.log.gz 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30858.log 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.30747.log.gz 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30580.log 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.31162.log: /var/log/ceph/ceph-client.admin.30858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-08-16T02:42:39.298 INFO:teuthology.orchestra.run.smithi116.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30858.log.gz 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30748.log 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.30580.log.gz 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32952.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32784.log 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30748.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30748.log.gz 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.30354.log 2024-08-16T02:42:39.299 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32910.log 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32784.log.gz 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30396.log 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32574.log 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 85.8% -- replaced with /var/log/ceph/ceph-client.admin.32910.log.gz 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.30396.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32490.log 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32574.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30128.log 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30396.log.gz 2024-08-16T02:42:39.300 INFO:teuthology.orchestra.run.smithi116.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.32574.log.gz 2024-08-16T02:42:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31631.log 2024-08-16T02:42:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.32490.log: /var/log/ceph/ceph-client.admin.30128.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.32490.log.gz 2024-08-16T02:42:39.301 INFO:teuthology.orchestra.run.smithi116.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.30128.log.gz 2024-08-16T02:42:39.301 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.31631.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.31162.log.gz 2024-08-16T02:42:39.304 INFO:teuthology.orchestra.run.smithi116.stderr: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.31631.log.gz 2024-08-16T02:42:39.313 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29399.log 2024-08-16T02:42:39.314 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-16T02:42:39.314 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29359.log 2024-08-16T02:42:39.314 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29399.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-16T02:42:39.314 INFO:teuthology.orchestra.run.smithi152.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.29399.log.gz 2024-08-16T02:42:39.315 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29359.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-16T02:42:39.315 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.3.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.29359.log.gz 2024-08-16T02:42:39.315 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29439.log 2024-08-16T02:42:39.316 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29319.log 2024-08-16T02:42:39.316 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29439.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.29439.log.gz 2024-08-16T02:42:39.316 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29319.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.29319.log.gz 2024-08-16T02:42:39.352 INFO:teuthology.orchestra.run.smithi152.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-16T02:42:39.538 INFO:teuthology.orchestra.run.smithi116.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-16T02:42:40.221 INFO:teuthology.orchestra.run.smithi152.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-16T02:42:40.249 INFO:teuthology.orchestra.run.smithi116.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-16T02:42:40.253 INFO:teuthology.orchestra.run.smithi152.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-16T02:42:40.255 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-08-16T02:42:40.255 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m0.952s 2024-08-16T02:42:40.255 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m1.810s 2024-08-16T02:42:40.255 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.096s 2024-08-16T02:42:40.410 INFO:teuthology.orchestra.run.smithi116.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-16T02:42:40.412 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-08-16T02:42:40.412 INFO:teuthology.orchestra.run.smithi116.stderr:real 0m1.146s 2024-08-16T02:42:40.412 INFO:teuthology.orchestra.run.smithi116.stderr:user 0m2.296s 2024-08-16T02:42:40.412 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m0.134s 2024-08-16T02:42:40.413 INFO:tasks.ceph:Archiving logs... 2024-08-16T02:42:40.413 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi116/log 2024-08-16T02:42:40.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:42:40.859 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi152/log 2024-08-16T02:42:40.860 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-16T02:42:41.160 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-16T02:42:41.169 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-16T02:42:41.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:42:41.172 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-16T02:42:41.269 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-16T02:42:41.277 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-16T02:42:41.277 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:42:41.280 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-16T02:42:41.292 INFO:teuthology.orchestra.run.smithi116.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:42:41.294 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:=============================================================================== 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 -26us[ -27us] +/- 41ms 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 13 -577us[ -578us] +/- 77ms 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +640us[ +639us] +/- 77ms 2024-08-16T02:42:41.297 INFO:teuthology.orchestra.run.smithi116.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:42:41.298 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-16T02:42:41.299 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-08-16T02:42:41.299 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv01.front.sepia.ceph.com 3 6 377 13 -19us[ -16us] +/- 41ms 2024-08-16T02:42:41.299 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 13 -565us[ -565us] +/- 77ms 2024-08-16T02:42:41.299 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +639us[ +639us] +/- 77ms 2024-08-16T02:42:41.299 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-08-16T02:42:41.300 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-16T02:42:41.323 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-16T02:42:41.323 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-16T02:42:41.375 DEBUG:teuthology.orchestra.run.smithi116:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:42:41.436 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-16T02:42:41.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:42:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:41.523 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-08-16T02:42:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:41.550 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-16T02:42:41.559 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-16T02:42:41.611 INFO:teuthology.task.internal:Duration was 809.182023 seconds 2024-08-16T02:42:41.611 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-16T02:42:41.661 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-16T02:42:41.661 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:42:41.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-16T02:42:41.700 INFO:teuthology.orchestra.run.smithi116.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:42:41.702 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-16T02:42:41.928 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-16T02:42:41.928 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-08-16T02:42:41.929 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:42:41.949 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-08-16T02:42:41.950 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-16T02:42:41.991 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-16T02:42:41.992 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:42:41.994 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-16T02:42:42.057 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-16T02:42:42.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:42.104 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:42.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-16T02:42:42.213 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-16T02:42:42.213 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:42:42.238 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-16T02:42:42.263 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-16T02:42:42.272 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:42.281 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:42.301 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-08-16T02:42:42.326 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-08-16T02:42:42.343 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:42.378 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi116.front.sepia.ceph.com, flagging run as failed 2024-08-16T02:42:42.378 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:42.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-16T02:42:42.407 INFO:teuthology.task.internal:Transferring archived files... 2024-08-16T02:42:42.408 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi116 2024-08-16T02:42:42.408 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-16T02:42:49.536 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2024-08-16T02:42:49.544 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yingxin-2024-08-16_01:59:29-crimson-rados-wip-yingxin-testing-2024-08-15-1551-distro-default-smithi/7857419/remote/smithi116/coredump/1723776131.31631.core.gz: gzip compressed data, was "1723776131.31631.core", last modified: Fri Aug 16 02:42:42 2024, max compression, original size modulo 2^32 139444224 2024-08-16T02:42:49.544 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2024-08-16T02:42:50.124 INFO:teuthology.task.internal: core looks like: /tmp/tmpad8tbata: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados_api_aio_pp', real uid: 1000, effective uid: 1000, real gid: 1316, effective gid: 1316, execfn: '/usr/bin/ceph_test_rados_api_aio_pp', platform: 'x86_64' 2024-08-16T02:42:50.137 INFO:teuthology.task.internal: dump_program: ceph_test_rados_api_aio_pp 2024-08-16T02:42:50.137 DEBUG:teuthology.orchestra.run.smithi116:> which ceph_test_rados_api_aio_pp 2024-08-16T02:42:50.224 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/bin/ceph_test_rados_api_aio_pp 2024-08-16T02:42:50.247 DEBUG:teuthology.orchestra.remote:smithi116:/usr/bin/ceph_test_rados_api_aio_pp is 1MB 2024-08-16T02:42:50.331 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-08-16T02:42:50.349 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.VEeJX6kaXn 2024-08-16T02:42:50.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.VEeJX6kaXn 2024-08-16T02:42:50.415 INFO:teuthology.orchestra.run.smithi116.stderr:cp: cannot stat '/usr/bin/ceph_test_rados_api_aio_pp.debug': No such file or directory 2024-08-16T02:42:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:50.417 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.VEeJX6kaXn' 2024-08-16T02:42:50.426 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-16T02:42:50.435 INFO:teuthology.task.internal:Not uploading archives. 2024-08-16T02:42:50.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-16T02:42:50.516 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-16T02:42:50.517 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:42:50.522 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-16T02:42:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: 262165 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest 2024-08-16T02:42:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: 405787 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:37 /home/ubuntu/cephtest/mnt.0 2024-08-16T02:42:50.537 INFO:teuthology.orchestra.run.smithi116.stdout: 405788 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:38 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 405832 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 262166 4 drwxr-xr-x 6 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 405786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 405790 120 -rw------- 1 ubuntu ubuntu 122631 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:42:50.538 INFO:teuthology.orchestra.run.smithi116.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:29 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-16T02:42:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: 265306 192 -rw-r--r-- 1 ubuntu ubuntu 192791 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: 265307 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:29 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-16T02:42:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: 262167 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/coredump 2024-08-16T02:42:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: 265470 86624 -rw------- 1 ubuntu ubuntu 139444224 Aug 16 02:42 /home/ubuntu/cephtest/archive/coredump/1723776131.31631.core 2024-08-16T02:42:50.539 INFO:teuthology.orchestra.run.smithi116.stdout: 409365 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:38 /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-16T02:42:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-16T02:42:50.540 INFO:teuthology.orchestra.run.smithi116.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:42:50.541 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 363, in fetch_binaries_for_coredumps remote.get_file(debug_path, coredump_path) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 622, in get_file self.run(args=args) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.VEeJX6kaXn' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi116 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-16T02:42:50.551 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-16T02:42:50.565 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-16T02:42:50.628 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi116 with status 1: 'sudo cp /usr/bin/ceph_test_rados_api_aio_pp.debug /tmp/tmp.VEeJX6kaXn' 2024-08-16T02:42:50.632 INFO:teuthology.run:Summary data: description: crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore tasks/rados_api_tests} duration: 809.1820225715637 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi116 with status 134: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6f386a3f0b1fe6a4d7219b54bf963d790bc84004 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b0dcd9f3e9c94a08b9c103a2da50edce status: fail success: false 2024-08-16T02:42:50.632 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 262165 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 262166 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 405786 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 406092 116 -rw------- 1 ubuntu ubuntu 117647 Aug 16 02:42 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 262168 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 262209 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:29 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2024-08-16T02:42:50.639 INFO:teuthology.orchestra.run.smithi152.stdout: 265306 188 -rw-r--r-- 1 ubuntu ubuntu 191810 Aug 16 02:42 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-16T02:42:50.640 INFO:teuthology.orchestra.run.smithi152.stdout: 265307 4 -rw-rw-rw- 1 ubuntu ubuntu 29 Aug 16 02:29 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2024-08-16T02:42:50.640 INFO:teuthology.orchestra.run.smithi152.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-16T02:42:50.722 INFO:teuthology.run:FAIL