2023-10-16T23:34:10.673 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-16T23:34:10.674 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} --archive /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682 --verbose --name sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi --owner scheduled_sjust@teuthology -- /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682/orig.config.yaml 2023-10-16T23:34:10.701 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-16T23:34:10.775 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682 branch: wip-crimson-scrub-testing-2023-10-15 description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} email: null first_in_suite: false job_id: '7430682' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-crimson-scrub-testing-2023-10-15 ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson seastar smp: 3 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 workunit: branch: wip-crimson-scrub-testing-2023-10-15 env: CRIMSON_COMPAT: '1' sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 owner: scheduled_sjust@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4255 sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-crimson-scrub-testing-2023-10-15 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeOtfSQGEi39/FCzuey/qkBwXw9MuQMnDkE0Bc7f2K1cUAKQ8bY00fJA8vzUJfVlCG/fRGvBKEk5YDeH430UO4= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - cls/test_cls_rbd.sh - cls/test_cls_lock.sh - cls/test_cls_journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_23:13:53 tube: smithi user: sjust verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-16T23:34:10.776 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa; will attempt to use it 2023-10-16T23:34:10.776 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks 2023-10-16T23:34:10.777 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-16T23:34:10.779 INFO:teuthology.task.internal:Checking packages... 2023-10-16T23:34:10.803 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '06f14dfbeeb0dce341775222d82ea9644e37c496' 2023-10-16T23:34:10.803 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-16T23:34:10.804 INFO:teuthology.packaging:ref: None 2023-10-16T23:34:10.804 INFO:teuthology.packaging:tag: None 2023-10-16T23:34:10.804 INFO:teuthology.packaging:branch: wip-crimson-scrub-testing-2023-10-15 2023-10-16T23:34:10.804 INFO:teuthology.packaging:sha1: 06f14dfbeeb0dce341775222d82ea9644e37c496 2023-10-16T23:34:10.805 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-crimson-scrub-testing-2023-10-15 2023-10-16T23:34:10.962 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6766.g06f14dfb 2023-10-16T23:34:10.962 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-16T23:34:10.968 INFO:teuthology.task.internal:no buildpackages task found 2023-10-16T23:34:10.968 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-16T23:34:11.070 INFO:teuthology.task.internal:Saving configuration 2023-10-16T23:34:11.082 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-16T23:34:11.137 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-16T23:34:11.165 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 23:29:45.616075', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeOtfSQGEi39/FCzuey/qkBwXw9MuQMnDkE0Bc7f2K1cUAKQ8bY00fJA8vzUJfVlCG/fRGvBKEk5YDeH430UO4='} 2023-10-16T23:34:11.166 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-16T23:34:11.187 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-16T23:34:11.188 INFO:teuthology.run_tasks:Running task console_log... 2023-10-16T23:34:11.269 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fca64f5b940>, signals=[15]) 2023-10-16T23:34:11.270 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-16T23:34:11.276 INFO:teuthology.task.internal:Opening connections... 2023-10-16T23:34:11.277 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2023-10-16T23:34:11.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T23:34:11.378 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-16T23:34:11.463 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2023-10-16T23:34:11.481 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2023-10-16T23:34:11.482 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2023-10-16T23:34:11.538 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2023-10-16T23:34:11.538 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2023-10-16T23:34:11.539 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2023-10-16T23:34:11.539 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2023-10-16T23:34:11.539 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2023-10-16T23:34:11.539 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2023-10-16T23:34:11.540 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-16T23:34:11.540 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2023-10-16T23:34:11.540 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2023-10-16T23:34:11.540 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-16T23:34:11.540 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2023-10-16T23:34:11.541 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-16T23:34:11.541 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-16T23:34:11.541 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-16T23:34:11.542 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-10-16T23:34:11.563 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-16T23:34:11.571 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-16T23:34:11.617 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-16T23:34:11.618 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2023-10-16T23:34:11.635 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-16T23:34:11.713 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-16T23:34:11.714 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2023-10-16T23:34:11.737 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-16T23:34:11.803 INFO:teuthology.run_tasks:Running task kernel... 2023-10-16T23:34:11.893 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-16T23:34:11.894 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-16T23:34:11.894 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-16T23:34:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T23:34:11.912 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2023-10-16T23:34:11.970 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-361.el9.x86_64 2023-10-16T23:34:11.971 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-361.el9.x86_64 2023-10-16T23:34:11.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2023-10-16T23:34:14.406 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.3 MB 00:00 2023-10-16T23:34:15.814 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-10-16T23:34:28.330 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 662 kB/s | 7.9 MB 00:12 2023-10-16T23:34:41.558 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 18 MB 00:12 2023-10-16T23:34:56.331 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 516 kB/s | 5.7 MB 00:11 2023-10-16T23:34:57.709 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 14 kB 00:00 2023-10-16T23:34:59.570 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-10-16T23:35:04.219 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2023-10-16T23:35:06.417 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-16T23:35:06.418 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-16T23:35:06.509 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:35:06.516 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:35:06.516 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:35:06.517 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:35:06.517 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:35:06.517 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-375.el9 CentOS-BaseOS 5.3 M 2023-10-16T23:35:06.517 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:35:06.518 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core x86_64 5.14.0-375.el9 CentOS-BaseOS 20 M 2023-10-16T23:35:06.518 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules x86_64 5.14.0-375.el9 CentOS-BaseOS 37 M 2023-10-16T23:35:06.518 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core x86_64 5.14.0-375.el9 CentOS-BaseOS 32 M 2023-10-16T23:35:06.518 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:06.519 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:35:06.519 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:35:06.519 INFO:teuthology.orchestra.run.smithi053.stdout:Install 4 Packages 2023-10-16T23:35:06.519 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:06.520 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 95 M 2023-10-16T23:35:06.520 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 120 M 2023-10-16T23:35:06.520 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:35:06.937 INFO:teuthology.orchestra.run.smithi053.stdout:(1/4): kernel-5.14.0-375.el9.x86_64.rpm 13 MB/s | 5.3 MB 00:00 2023-10-16T23:35:07.971 INFO:teuthology.orchestra.run.smithi053.stdout:(2/4): kernel-core-5.14.0-375.el9.x86_64.rpm 14 MB/s | 20 MB 00:01 2023-10-16T23:35:08.313 INFO:teuthology.orchestra.run.smithi053.stdout:(3/4): kernel-modules-core-5.14.0-375.el9.x86_6 23 MB/s | 32 MB 00:01 2023-10-16T23:35:08.654 INFO:teuthology.orchestra.run.smithi053.stdout:(4/4): kernel-modules-5.14.0-375.el9.x86_64.rpm 18 MB/s | 37 MB 00:02 2023-10-16T23:35:08.658 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:35:08.658 INFO:teuthology.orchestra.run.smithi053.stdout:Total 44 MB/s | 95 MB 00:02 2023-10-16T23:35:08.658 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:35:09.090 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:35:09.090 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:35:10.127 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:35:10.127 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:35:12.651 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:35:13.311 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-core-5.14.0-375.el9.x86_64 1/4 2023-10-16T23:35:13.438 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-16T23:35:16.283 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-16T23:35:16.458 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-16T23:35:21.394 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-16T23:35:21.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : kernel-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:26.466 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:57.189 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-core-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:57.234 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-modules-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:57.667 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:57.667 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-375.el9.x86_64 1/4 2023-10-16T23:35:57.668 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-core-5.14.0-375.el9.x86_64 2/4 2023-10-16T23:35:57.668 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-5.14.0-375.el9.x86_64 3/4 2023-10-16T23:35:58.105 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-modules-core-5.14.0-375.el9.x86_64 4/4 2023-10-16T23:35:58.105 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:58.106 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:35:58.106 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.106 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-core-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.106 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.107 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-modules-core-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.107 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:58.107 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:35:58.592 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T23:35:58.719 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.720 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-16T23:35:58.720 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-375.el9.x86_64 2023-10-16T23:35:58.720 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-16T23:35:58.721 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-16T23:35:58.721 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-16T23:35:58.721 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=None, version=distro) 2023-10-16T23:35:58.721 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2023-10-16T23:35:58.830 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-375.el9.x86_64 2023-10-16T23:35:58.831 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-375.el9.x86_64 2023-10-16T23:35:58.831 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2023-10-16T23:35:58.880 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2023-10-16T23:35:58.880 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2023-10-16T23:35:58.880 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2023-10-16T23:35:58.880 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2023-10-16T23:35:58.881 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2023-10-16T23:35:58.881 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-16T23:35:58.881 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2023-10-16T23:35:58.881 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2023-10-16T23:35:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2023-10-16T23:35:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-16T23:35:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-16T23:35:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-16T23:35:58.882 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-16T23:35:58.883 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2023-10-16T23:35:58.883 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2023-10-16T23:35:58.883 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-16T23:35:58.883 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2023-10-16T23:35:58.883 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2023-10-16T23:35:58.884 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:58.884 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-16T23:35:58.884 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-16T23:35:58.884 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-16T23:35:58.884 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2023-10-16T23:35:58.885 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:35:58.885 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2023-10-16T23:35:58.886 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-16T23:35:58.886 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 5.14.0-375.el9.x86_64 2023-10-16T23:35:58.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-16T23:35:59.426 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2023-10-16T23:36:00.834 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-16T23:36:00.877 INFO:teuthology.orchestra.run.smithi053.stderr:done 2023-10-16T23:36:00.879 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2023-10-16T23:36:00.895 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.6ii1mDpYSv 2023-10-16T23:36:00.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6ii1mDpYSv 2023-10-16T23:36:00.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.6ii1mDpYSv 2023-10-16T23:36:01.113 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.6ii1mDpYSv is 6KB 2023-10-16T23:36:01.162 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.6ii1mDpYSv 2023-10-16T23:36:01.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2023-10-16T23:36:01.254 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-16T23:36:01.255 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-16T23:36:01.255 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-16T23:36:01.255 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64.conf 2023-10-16T23:36:01.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-375.el9.x86_64 2023-10-16T23:36:01.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2023-10-16T23:36:31.446 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-16T23:36:31.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-10-16T23:36:31.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T23:37:31.507 DEBUG:teuthology.orchestra.remote:timed out 2023-10-16T23:37:40.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-10-16T23:37:40.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T23:37:40.921 DEBUG:teuthology.orchestra.run.smithi053:> true 2023-10-16T23:37:41.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-10-16T23:37:41.295 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-375.el9.x86_64"... 2023-10-16T23:37:41.295 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2023-10-16T23:37:41.310 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-375.el9.x86_64 2023-10-16T23:37:41.311 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-375.el9.x86_64 vs 5.14.0-375.el9.x86_64 2023-10-16T23:37:41.311 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-16T23:37:41.311 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-16T23:37:42.312 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-16T23:37:42.313 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-16T23:37:42.394 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2023-10-16T23:37:42.425 DEBUG:teuthology.parallel:result is None 2023-10-16T23:37:42.426 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-16T23:37:42.434 INFO:teuthology.task.internal:Creating test directory... 2023-10-16T23:37:42.434 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-16T23:37:42.453 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-16T23:37:42.483 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-16T23:37:42.580 INFO:teuthology.task.internal:Creating archive directory... 2023-10-16T23:37:42.581 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-16T23:37:42.722 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-16T23:37:42.729 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-16T23:37:42.730 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-16T23:37:42.772 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T23:37:42.790 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-16T23:37:42.791 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-16T23:37:42.798 INFO:teuthology.task.internal:Configuring sudo... 2023-10-16T23:37:42.798 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-16T23:37:42.865 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-16T23:37:42.874 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-16T23:37:42.875 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-16T23:37:42.922 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T23:37:43.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-16T23:37:43.059 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T23:37:43.141 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-16T23:37:43.190 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:37:43.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-16T23:37:43.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2023-10-16T23:37:43.355 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-16T23:37:43.641 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-16T23:37:43.649 INFO:teuthology.task.internal:Starting timer... 2023-10-16T23:37:43.650 INFO:teuthology.run_tasks:Running task pcp... 2023-10-16T23:37:43.745 INFO:teuthology.run_tasks:Running task selinux... 2023-10-16T23:37:43.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2023-10-16T23:37:43.868 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2023-10-16T23:37:43.870 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-16T23:37:43.870 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2023-10-16T23:37:43.900 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2023-10-16T23:37:43.900 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive'} 2023-10-16T23:37:43.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-16T23:37:43.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T23:37:43.971 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-16T23:37:43.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2023-10-16T23:37:44.052 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-16T23:37:44.062 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-16T23:37:44.068 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-16T23:37:44.177 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-16T23:37:44.194 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-16T23:37:44.195 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-16T23:43:15.633 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2023-10-16T23:43:15.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-10-16T23:43:15.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-16T23:43:15.734 DEBUG:teuthology.orchestra.run.smithi053:> true 2023-10-16T23:43:15.801 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-10-16T23:43:15.802 INFO:teuthology.run_tasks:Running task clock... 2023-10-16T23:43:15.813 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-16T23:43:15.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-16T23:43:15.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-16T23:43:15.875 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-16T23:43:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-16T23:43:15.931 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2023-10-16T23:43:15.949 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2023-10-16T23:43:15.968 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-16T23:43:15.988 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-16T23:43:16.036 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2023-10-16T23:43:16.039 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-16T23:43:16.039 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2023-10-16T23:43:16.040 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T23:43:16.040 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T23:43:16.040 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T23:43:16.040 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-16T23:43:16.041 INFO:teuthology.run_tasks:Running task install... 2023-10-16T23:43:16.049 DEBUG:teuthology.task.install:project ceph 2023-10-16T23:43:16.049 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496'}} 2023-10-16T23:43:16.050 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496'} 2023-10-16T23:43:16.050 INFO:teuthology.task.install:Using flavor: crimson 2023-10-16T23:43:16.056 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-16T23:43:16.056 INFO:teuthology.task.install:extra packages: [] 2023-10-16T23:43:16.057 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496', 'tag': None, 'wait_for_package': False} 2023-10-16T23:43:16.057 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=06f14dfbeeb0dce341775222d82ea9644e37c496 2023-10-16T23:43:16.314 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-15/06f14dfbeeb0dce341775222d82ea9644e37c496/centos/9/flavors/crimson/ 2023-10-16T23:43:16.315 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6766.g06f14dfb 2023-10-16T23:43:16.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-15/06f14dfbeeb0dce341775222d82ea9644e37c496/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-15/06f14dfbeeb0dce341775222d82ea9644e37c496/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-crimson-scrub-testing-2023-10-15/06f14dfbeeb0dce341775222d82ea9644e37c496/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-16T23:43:16.580 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:43:16.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-16T23:43:16.613 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-16T23:43:16.613 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/06f14dfbeeb0dce341775222d82ea9644e37c496/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-16T23:43:16.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-16T23:43:16.772 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-16T23:43:16.802 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2023-10-16T23:43:16.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2023-10-16T23:43:17.057 INFO:teuthology.orchestra.run.smithi053.stdout:72 files removed 2023-10-16T23:43:17.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw 2023-10-16T23:43:17.838 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 193 kB/s | 81 kB 00:00 2023-10-16T23:43:18.222 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-10-16T23:43:18.639 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2023-10-16T23:43:19.175 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 13 MB/s | 6.3 MB 00:00 2023-10-16T23:43:20.584 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-10-16T23:43:33.309 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 650 kB/s | 7.9 MB 00:12 2023-10-16T23:43:46.330 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 18 MB 00:11 2023-10-16T23:44:02.045 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 481 kB/s | 5.7 MB 00:12 2023-10-16T23:44:03.799 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 14 kB 00:00 2023-10-16T23:44:05.920 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 19 MB 00:02 2023-10-16T23:44:10.559 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-10-16T23:44:12.975 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:44:12.986 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:12.987 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:44:12.987 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:12.987 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:44:12.987 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 8.4 M 2023-10-16T23:44:12.988 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2023-10-16T23:44:12.988 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 3.1 M 2023-10-16T23:44:12.988 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 3.0 M 2023-10-16T23:44:12.988 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:44:12.989 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options 2023-10-16T23:44:12.989 INFO:teuthology.orchestra.run.smithi053.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-16T23:44:12.989 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 6.6 M 2023-10-16T23:44:12.989 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 17 M 2023-10-16T23:44:12.989 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 24 k 2023-10-16T23:44:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-16T23:44:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-16T23:44:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-16T23:44:12.990 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-16T23:44:12.991 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 726 k 2023-10-16T23:44:12.991 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-16T23:44:12.991 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-16T23:44:12.991 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-16T23:44:12.991 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 470 k 2023-10-16T23:44:12.992 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-16T23:44:12.992 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 4.5 M 2023-10-16T23:44:12.992 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-16T23:44:12.992 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-16T23:44:12.992 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 45 k 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 127 k 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 162 k 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-16T23:44:12.993 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 320 k 2023-10-16T23:44:12.994 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 298 k 2023-10-16T23:44:12.994 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 99 k 2023-10-16T23:44:12.994 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-16T23:44:12.994 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-16T23:44:12.995 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:12.995 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:44:12.995 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:12.995 INFO:teuthology.orchestra.run.smithi053.stdout:Install 28 Packages 2023-10-16T23:44:12.995 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 2 Packages 2023-10-16T23:44:12.996 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:12.996 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 54 M 2023-10-16T23:44:12.997 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:44:13.968 INFO:teuthology.orchestra.run.smithi053.stdout:(1/30): ceph-base-18.0.0-6766.g06f14dfb.el9.x86 9.0 MB/s | 6.6 MB 00:00 2023-10-16T23:44:14.044 INFO:teuthology.orchestra.run.smithi053.stdout:(2/30): ceph-selinux-18.0.0-6766.g06f14dfb.el9. 321 kB/s | 24 kB 00:00 2023-10-16T23:44:14.144 INFO:teuthology.orchestra.run.smithi053.stdout:(3/30): libcephfs2-18.0.0-6766.g06f14dfb.el9.x8 7.1 MB/s | 726 kB 00:00 2023-10-16T23:44:14.252 INFO:teuthology.orchestra.run.smithi053.stdout:(4/30): libradosstriper1-18.0.0-6766.g06f14dfb. 4.2 MB/s | 470 kB 00:00 2023-10-16T23:44:14.545 INFO:teuthology.orchestra.run.smithi053.stdout:(5/30): ceph-radosgw-18.0.0-6766.g06f14dfb.el9. 6.4 MB/s | 8.4 MB 00:01 2023-10-16T23:44:14.628 INFO:teuthology.orchestra.run.smithi053.stdout:(6/30): python3-ceph-argparse-18.0.0-6766.g06f1 542 kB/s | 45 kB 00:00 2023-10-16T23:44:14.695 INFO:teuthology.orchestra.run.smithi053.stdout:(7/30): librgw2-18.0.0-6766.g06f14dfb.el9.x86_6 10 MB/s | 4.5 MB 00:00 2023-10-16T23:44:14.720 INFO:teuthology.orchestra.run.smithi053.stdout:(8/30): python3-ceph-common-18.0.0-6766.g06f14d 1.3 MB/s | 127 kB 00:00 2023-10-16T23:44:14.754 INFO:teuthology.orchestra.run.smithi053.stdout:(9/30): python3-cephfs-18.0.0-6766.g06f14dfb.el 2.7 MB/s | 162 kB 00:00 2023-10-16T23:44:14.788 INFO:teuthology.orchestra.run.smithi053.stdout:(10/30): python3-rados-18.0.0-6766.g06f14dfb.el 4.7 MB/s | 320 kB 00:00 2023-10-16T23:44:14.821 INFO:teuthology.orchestra.run.smithi053.stdout:(11/30): python3-rbd-18.0.0-6766.g06f14dfb.el9. 4.3 MB/s | 298 kB 00:00 2023-10-16T23:44:14.847 INFO:teuthology.orchestra.run.smithi053.stdout:(12/30): python3-rgw-18.0.0-6766.g06f14dfb.el9. 1.7 MB/s | 99 kB 00:00 2023-10-16T23:44:15.022 INFO:teuthology.orchestra.run.smithi053.stdout:(13/30): ceph-common-18.0.0-6766.g06f14dfb.el9. 9.6 MB/s | 17 MB 00:01 2023-10-16T23:44:15.164 INFO:teuthology.orchestra.run.smithi053.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 309 kB/s | 106 kB 00:00 2023-10-16T23:44:15.189 INFO:teuthology.orchestra.run.smithi053.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 465 kB/s | 159 kB 00:00 2023-10-16T23:44:15.223 INFO:teuthology.orchestra.run.smithi053.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2023-10-16T23:44:15.248 INFO:teuthology.orchestra.run.smithi053.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 710 kB/s | 41 kB 00:00 2023-10-16T23:44:15.307 INFO:teuthology.orchestra.run.smithi053.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 661 kB 00:00 2023-10-16T23:44:15.332 INFO:teuthology.orchestra.run.smithi053.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2023-10-16T23:44:15.899 INFO:teuthology.orchestra.run.smithi053.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 170 kB/s | 111 kB 00:00 2023-10-16T23:44:16.007 INFO:teuthology.orchestra.run.smithi053.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 225 kB/s | 24 kB 00:00 2023-10-16T23:44:16.092 INFO:teuthology.orchestra.run.smithi053.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 394 kB/s | 309 kB 00:00 2023-10-16T23:44:16.117 INFO:teuthology.orchestra.run.smithi053.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2023-10-16T23:44:16.201 INFO:teuthology.orchestra.run.smithi053.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2023-10-16T23:44:16.476 INFO:teuthology.orchestra.run.smithi053.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 697 kB/s | 192 kB 00:00 2023-10-16T23:44:16.592 INFO:teuthology.orchestra.run.smithi053.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 1.7 MB/s | 837 kB 00:00 2023-10-16T23:44:16.851 INFO:teuthology.orchestra.run.smithi053.stdout:(27/30): librados2-18.0.0-6766.g06f14dfb.el9.x8 12 MB/s | 3.1 MB 00:00 2023-10-16T23:44:16.996 INFO:teuthology.orchestra.run.smithi053.stdout:(28/30): libarrow-9.0.0-9.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2023-10-16T23:44:17.055 INFO:teuthology.orchestra.run.smithi053.stdout:(29/30): librbd1-18.0.0-6766.g06f14dfb.el9.x86_ 15 MB/s | 3.0 MB 00:00 2023-10-16T23:44:18.555 INFO:teuthology.orchestra.run.smithi053.stdout:(30/30): thrift-0.15.0-2.el9.x86_64.rpm 779 kB/s | 1.6 MB 00:02 2023-10-16T23:44:18.562 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:44:18.563 INFO:teuthology.orchestra.run.smithi053.stdout:Total 9.7 MB/s | 54 MB 00:05 2023-10-16T23:44:18.575 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:44:18.617 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:44:18.617 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:44:19.097 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:44:19.098 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:44:20.525 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:44:20.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-16T23:44:20.992 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/32 2023-10-16T23:44:21.083 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/32 2023-10-16T23:44:21.182 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:18.0.0-6766.g06f14dfb.el9.x86_64 3/32 2023-10-16T23:44:21.217 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:18.0.0-6766.g06f14dfb.el9.x86_64 4/32 2023-10-16T23:44:21.280 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:18.0.0-6766.g06f14dfb.el9.x86_64 4/32 2023-10-16T23:44:21.341 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-16T23:44:21.380 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-16T23:44:21.418 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-16T23:44:21.462 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:18.0.0-6766.g06f14dfb.el9. 8/32 2023-10-16T23:44:21.539 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:18.0.0-6766.g06f14dfb.el9.x86_64 9/32 2023-10-16T23:44:21.563 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:18.0.0-6766.g06f14dfb.el9.x86_6 10/32 2023-10-16T23:44:21.635 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6766.g06f14dfb.el9.x86_6 10/32 2023-10-16T23:44:21.670 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-16T23:44:21.780 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-16T23:44:21.821 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-16T23:44:21.862 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-16T23:44:21.894 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-16T23:44:21.921 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-16T23:44:21.959 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-16T23:44:21.992 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-16T23:44:22.321 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-16T23:44:22.346 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:18.0.0-6766.g06f14dfb.el9.x86_64 20/32 2023-10-16T23:44:22.421 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:18.0.0-6766.g06f14dfb.el9.x86_64 20/32 2023-10-16T23:44:22.451 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 21/32 2023-10-16T23:44:22.487 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-16T23:44:22.941 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-16T23:44:22.974 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:18.0.0-6766.g06f14dfb.el9.x86_64 24/32 2023-10-16T23:44:23.045 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:18.0.0-6766.g06f14dfb.el9.x86_64 24/32 2023-10-16T23:44:23.124 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 25/32 2023-10-16T23:44:23.151 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:18.0.0-6766.g06f14dfb.el9.x8 26/32 2023-10-16T23:44:25.199 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 27/32 2023-10-16T23:44:25.239 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 27/32 2023-10-16T23:44:26.010 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 27/32 2023-10-16T23:44:26.050 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:18.0.0-6766.g06f14dfb.el9.x86_64 28/32 2023-10-16T23:44:26.122 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:18.0.0-6766.g06f14dfb.el9.x86_64 28/32 2023-10-16T23:44:26.122 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-16T23:44:26.122 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-16T23:44:26.122 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:26.160 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:18.0.0-6766.g06f14dfb.el9.x86_64 29/32 2023-10-16T23:44:36.512 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6766.g06f14dfb.el9.x86_64 29/32 2023-10-16T23:44:36.512 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2023-10-16T23:44:36.512 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2023-10-16T23:44:36.512 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2023-10-16T23:44:36.513 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2023-10-16T23:44:36.513 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2023-10-16T23:44:36.513 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2023-10-16T23:44:36.513 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2023-10-16T23:44:36.513 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:37.392 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 30/32 2023-10-16T23:44:37.417 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 30/32 2023-10-16T23:44:37.417 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:44:37.418 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-16T23:44:37.418 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T23:44:37.418 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-16T23:44:37.418 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:37.419 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-16T23:44:37.494 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-16T23:44:37.495 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-16T23:44:38.677 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-16T23:44:38.678 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/32 2023-10-16T23:44:38.678 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/32 2023-10-16T23:44:38.678 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 3/32 2023-10-16T23:44:38.679 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:18.0.0-6766.g06f14dfb.el9.x86_64 4/32 2023-10-16T23:44:38.679 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:18.0.0-6766.g06f14dfb.el9.x86_64 5/32 2023-10-16T23:44:38.679 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:18.0.0-6766.g06f14dfb.el9.x86_6 6/32 2023-10-16T23:44:38.679 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:18.0.0-6766.g06f14dfb.el9.x86_64 7/32 2023-10-16T23:44:38.679 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6766.g06f14dfb.el9. 8/32 2023-10-16T23:44:38.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:18.0.0-6766.g06f14dfb.el9.x8 9/32 2023-10-16T23:44:38.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:18.0.0-6766.g06f14dfb.el9.x86_64 10/32 2023-10-16T23:44:38.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:18.0.0-6766.g06f14dfb.el9.x86_64 11/32 2023-10-16T23:44:38.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 12/32 2023-10-16T23:44:38.680 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 13/32 2023-10-16T23:44:38.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-16T23:44:38.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-16T23:44:38.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-16T23:44:38.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-16T23:44:38.681 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-16T23:44:38.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-16T23:44:38.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-16T23:44:38.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-16T23:44:38.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-16T23:44:38.682 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-16T23:44:38.683 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-16T23:44:38.683 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-16T23:44:38.683 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-16T23:44:38.683 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-16T23:44:38.683 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-16T23:44:38.684 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:18.0.0-6766.g06f14dfb.el9.x86_64 29/32 2023-10-16T23:44:38.684 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-16T23:44:38.684 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:18.0.0-6766.g06f14dfb.el9.x86_64 31/32 2023-10-16T23:44:39.822 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-16T23:44:39.822 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:39.823 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2023-10-16T23:44:39.823 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.823 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.823 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:44:39.823 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-16T23:44:39.824 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.824 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.824 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.824 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.824 INFO:teuthology.orchestra.run.smithi053.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-16T23:44:39.825 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-16T23:44:39.825 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-16T23:44:39.825 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-16T23:44:39.825 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.825 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-16T23:44:39.826 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-16T23:44:39.826 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-16T23:44:39.826 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.826 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-16T23:44:39.826 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.827 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-16T23:44:39.827 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-16T23:44:39.827 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-16T23:44:39.827 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.827 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-16T23:44:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.828 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:44:39.829 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2023-10-16T23:44:39.829 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-16T23:44:39.829 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:39.829 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:44:40.157 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-test 2023-10-16T23:44:40.781 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:30 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:44:40.934 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:44:40.935 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:40.935 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:44:40.935 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:40.935 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:44:40.936 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 117 M 2023-10-16T23:44:40.936 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:44:40.936 INFO:teuthology.orchestra.run.smithi053.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-16T23:44:40.936 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-16T23:44:40.936 INFO:teuthology.orchestra.run.smithi053.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 161 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-16T23:44:40.937 INFO:teuthology.orchestra.run.smithi053.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-16T23:44:40.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:40.938 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:44:40.938 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:44:40.938 INFO:teuthology.orchestra.run.smithi053.stdout:Install 10 Packages 2023-10-16T23:44:40.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:44:40.939 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 120 M 2023-10-16T23:44:40.939 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.1 G 2023-10-16T23:44:40.940 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:44:41.256 INFO:teuthology.orchestra.run.smithi053.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.3 MB/s | 418 kB 00:00 2023-10-16T23:44:41.298 INFO:teuthology.orchestra.run.smithi053.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2023-10-16T23:44:41.340 INFO:teuthology.orchestra.run.smithi053.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-10-16T23:44:41.390 INFO:teuthology.orchestra.run.smithi053.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2023-10-16T23:44:41.416 INFO:teuthology.orchestra.run.smithi053.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2023-10-16T23:44:41.466 INFO:teuthology.orchestra.run.smithi053.stdout:(6/10): libcephsqlite-18.0.0-6766.g06f14dfb.el9 412 kB/s | 161 kB 00:00 2023-10-16T23:44:41.557 INFO:teuthology.orchestra.run.smithi053.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-10-16T23:44:41.691 INFO:teuthology.orchestra.run.smithi053.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 913 kB/s | 122 kB 00:00 2023-10-16T23:44:41.766 INFO:teuthology.orchestra.run.smithi053.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 4.2 MB/s | 1.3 MB 00:00 2023-10-16T23:44:46.283 INFO:teuthology.orchestra.run.smithi053.stdout:(10/10): ceph-test-18.0.0-6766.g06f14dfb.el9.x8 23 MB/s | 117 MB 00:05 2023-10-16T23:44:46.286 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:44:46.286 INFO:teuthology.orchestra.run.smithi053.stdout:Total 22 MB/s | 120 MB 00:05 2023-10-16T23:44:46.301 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:44:46.325 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:44:46.326 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:44:47.202 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:44:47.203 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:44:48.336 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:44:48.426 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-16T23:44:48.463 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-16T23:44:48.503 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-16T23:44:48.550 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-16T23:44:48.585 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-16T23:44:48.616 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-16T23:44:48.655 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-16T23:44:48.761 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-16T23:44:48.788 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:18.0.0-6766.g06f14dfb.el9.x86_64 9/10 2023-10-16T23:45:05.970 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6766.g06f14dfb.el9.x86_64 9/10 2023-10-16T23:45:06.054 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:18.0.0-6766.g06f14dfb.el9.x86_64 10/10 2023-10-16T23:45:14.259 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-test-2:18.0.0-6766.g06f14dfb.el9.x86_64 10/10 2023-10-16T23:45:14.259 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/10 2023-10-16T23:45:14.259 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/10 2023-10-16T23:45:14.259 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-16T23:45:14.260 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-16T23:45:14.260 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-16T23:45:14.260 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-16T23:45:14.260 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-16T23:45:14.261 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-16T23:45:14.261 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-16T23:45:14.769 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-16T23:45:14.770 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:14.770 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:45:14.770 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:45:14.770 INFO:teuthology.orchestra.run.smithi053.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-16T23:45:14.770 INFO:teuthology.orchestra.run.smithi053.stdout: jq-1.6-15.el9.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-16T23:45:14.771 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-16T23:45:14.772 INFO:teuthology.orchestra.run.smithi053.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-16T23:45:14.772 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:14.772 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:45:15.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph 2023-10-16T23:45:15.716 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:05 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:45:15.867 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:45:15.869 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2023-10-16T23:45:15.870 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:45:15.870 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2023-10-16T23:45:15.870 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:45:15.870 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 6.5 k 2023-10-16T23:45:15.871 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:45:15.871 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 2.0 M 2023-10-16T23:45:15.871 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 1.5 M 2023-10-16T23:45:15.871 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 242 k 2023-10-16T23:45:15.871 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 8.8 M 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 118 M 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-16T23:45:15.872 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-16T23:45:15.873 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-16T23:45:15.873 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-16T23:45:15.873 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-16T23:45:15.873 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-16T23:45:15.873 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-16T23:45:15.874 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-16T23:45:15.874 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-16T23:45:15.874 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-16T23:45:15.874 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-16T23:45:15.874 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-16T23:45:15.875 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-16T23:45:15.876 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-16T23:45:15.876 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-16T23:45:15.876 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-16T23:45:15.876 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-16T23:45:15.876 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout:========================================================================================= 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout:Install 33 Packages 2023-10-16T23:45:15.877 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:15.878 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 135 M 2023-10-16T23:45:15.878 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.4 G 2023-10-16T23:45:15.878 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:45:16.443 INFO:teuthology.orchestra.run.smithi053.stdout:(1/33): ceph-18.0.0-6766.g06f14dfb.el9.x86_64.r 26 kB/s | 6.5 kB 00:00 2023-10-16T23:45:16.735 INFO:teuthology.orchestra.run.smithi053.stdout:(2/33): ceph-mds-18.0.0-6766.g06f14dfb.el9.x86_ 3.6 MB/s | 2.0 MB 00:00 2023-10-16T23:45:16.993 INFO:teuthology.orchestra.run.smithi053.stdout:(3/33): ceph-mgr-18.0.0-6766.g06f14dfb.el9.x86_ 1.8 MB/s | 1.5 MB 00:00 2023-10-16T23:45:17.111 INFO:teuthology.orchestra.run.smithi053.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6766.g06f1 2.0 MB/s | 242 kB 00:00 2023-10-16T23:45:17.310 INFO:teuthology.orchestra.run.smithi053.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 721 kB/s | 144 kB 00:00 2023-10-16T23:45:17.352 INFO:teuthology.orchestra.run.smithi053.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 762 kB/s | 32 kB 00:00 2023-10-16T23:45:17.386 INFO:teuthology.orchestra.run.smithi053.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-10-16T23:45:17.436 INFO:teuthology.orchestra.run.smithi053.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-10-16T23:45:17.478 INFO:teuthology.orchestra.run.smithi053.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2023-10-16T23:45:17.520 INFO:teuthology.orchestra.run.smithi053.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-10-16T23:45:17.562 INFO:teuthology.orchestra.run.smithi053.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2023-10-16T23:45:17.604 INFO:teuthology.orchestra.run.smithi053.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-10-16T23:45:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 3.1 MB/s | 1.2 MB 00:00 2023-10-16T23:45:18.096 INFO:teuthology.orchestra.run.smithi053.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 512 kB/s | 51 kB 00:00 2023-10-16T23:45:18.497 INFO:teuthology.orchestra.run.smithi053.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 109 kB/s | 43 kB 00:00 2023-10-16T23:45:18.947 INFO:teuthology.orchestra.run.smithi053.stdout:(16/33): ceph-mon-18.0.0-6766.g06f14dfb.el9.x86 3.5 MB/s | 8.8 MB 00:02 2023-10-16T23:45:18.972 INFO:teuthology.orchestra.run.smithi053.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 362 kB/s | 172 kB 00:00 2023-10-16T23:45:19.089 INFO:teuthology.orchestra.run.smithi053.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 92 kB/s | 11 kB 00:00 2023-10-16T23:45:19.214 INFO:teuthology.orchestra.run.smithi053.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2023-10-16T23:45:19.339 INFO:teuthology.orchestra.run.smithi053.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2023-10-16T23:45:19.465 INFO:teuthology.orchestra.run.smithi053.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 156 kB/s | 19 kB 00:00 2023-10-16T23:45:19.590 INFO:teuthology.orchestra.run.smithi053.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2023-10-16T23:45:19.707 INFO:teuthology.orchestra.run.smithi053.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 397 kB/s | 46 kB 00:00 2023-10-16T23:45:20.041 INFO:teuthology.orchestra.run.smithi053.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 328 kB/s | 358 kB 00:01 2023-10-16T23:45:20.066 INFO:teuthology.orchestra.run.smithi053.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 220 kB/s | 79 kB 00:00 2023-10-16T23:45:20.191 INFO:teuthology.orchestra.run.smithi053.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 132 kB/s | 16 kB 00:00 2023-10-16T23:45:20.417 INFO:teuthology.orchestra.run.smithi053.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 399 kB/s | 90 kB 00:00 2023-10-16T23:45:20.559 INFO:teuthology.orchestra.run.smithi053.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 252 kB/s | 36 kB 00:00 2023-10-16T23:45:20.584 INFO:teuthology.orchestra.run.smithi053.stdout:(29/33): python3-pecan-1.4.2-3.el9.noarch.rpm 501 kB/s | 272 kB 00:00 2023-10-16T23:45:21.001 INFO:teuthology.orchestra.run.smithi053.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 520 kB/s | 230 kB 00:00 2023-10-16T23:45:21.118 INFO:teuthology.orchestra.run.smithi053.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2023-10-16T23:45:21.309 INFO:teuthology.orchestra.run.smithi053.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 589 kB/s | 427 kB 00:00 2023-10-16T23:45:26.184 INFO:teuthology.orchestra.run.smithi053.stdout:(33/33): ceph-osd-18.0.0-6766.g06f14dfb.el9.x86 13 MB/s | 118 MB 00:09 2023-10-16T23:45:26.188 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:45:26.188 INFO:teuthology.orchestra.run.smithi053.stdout:Total 13 MB/s | 135 MB 00:10 2023-10-16T23:45:26.222 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:45:26.265 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:45:26.265 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:45:27.301 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:45:27.301 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:45:29.211 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:45:29.270 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-16T23:45:29.302 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-16T23:45:29.336 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-16T23:45:29.430 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-16T23:45:29.515 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-16T23:45:29.563 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-16T23:45:29.601 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-16T23:45:29.679 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-16T23:45:29.732 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-10-16T23:45:29.770 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-16T23:45:29.820 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-16T23:45:29.885 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-16T23:45:30.094 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-16T23:45:30.140 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-16T23:45:30.179 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-16T23:45:30.213 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-16T23:45:30.243 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-16T23:45:30.322 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-16T23:45:30.362 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-16T23:45:30.395 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-16T23:45:30.446 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-16T23:45:30.480 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-16T23:45:30.572 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-16T23:45:30.611 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-16T23:45:30.685 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-16T23:45:30.825 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-16T23:45:30.958 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-16T23:45:31.126 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6766.g06f14dfb.el9. 28/33 2023-10-16T23:45:31.157 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:18.0.0-6766.g06f14dfb.el9.x86_64 29/33 2023-10-16T23:45:31.182 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6766.g06f14dfb.el9.x86_64 29/33 2023-10-16T23:45:31.183 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:45:31.183 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-16T23:45:31.183 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T23:45:31.183 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-16T23:45:31.184 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:50.839 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:18.0.0-6766.g06f14dfb.el9.x86_64 30/33 2023-10-16T23:45:50.867 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:18.0.0-6766.g06f14dfb.el9.x86_64 30/33 2023-10-16T23:45:50.867 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:45:50.868 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-16T23:45:50.868 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T23:45:50.868 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-16T23:45:50.868 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:51.798 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:18.0.0-6766.g06f14dfb.el9.x86_64 31/33 2023-10-16T23:45:51.823 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:18.0.0-6766.g06f14dfb.el9.x86_64 31/33 2023-10-16T23:45:51.823 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:45:51.823 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-16T23:45:51.823 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T23:45:51.824 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-16T23:45:51.824 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:52.041 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:18.0.0-6766.g06f14dfb.el9.x86_64 32/33 2023-10-16T23:45:52.065 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:18.0.0-6766.g06f14dfb.el9.x86_64 32/33 2023-10-16T23:45:52.065 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:45:52.066 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-16T23:45:52.066 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T23:45:52.066 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-16T23:45:52.066 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:45:52.132 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:18.0.0-6766.g06f14dfb.el9.x86_64 33/33 2023-10-16T23:46:02.090 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-2:18.0.0-6766.g06f14dfb.el9.x86_64 33/33 2023-10-16T23:46:02.090 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/33 2023-10-16T23:46:02.090 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/33 2023-10-16T23:46:02.091 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:18.0.0-6766.g06f14dfb.el9.x86_64 3/33 2023-10-16T23:46:02.091 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:18.0.0-6766.g06f14dfb.el9.x86_64 4/33 2023-10-16T23:46:02.091 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:18.0.0-6766.g06f14dfb.el9.x86_64 5/33 2023-10-16T23:46:02.091 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6766.g06f14dfb.el9. 6/33 2023-10-16T23:46:02.091 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-16T23:46:02.092 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-16T23:46:02.093 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-16T23:46:02.094 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-16T23:46:02.094 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-16T23:46:02.094 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-16T23:46:02.094 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-16T23:46:02.094 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-16T23:46:02.095 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-16T23:46:02.095 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-16T23:46:02.095 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-16T23:46:02.096 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-16T23:46:02.096 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-16T23:46:02.096 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-16T23:46:02.096 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-16T23:46:02.096 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-16T23:46:03.051 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-16T23:46:03.051 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:03.051 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:03.051 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:03.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:03.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:03.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:03.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:03.052 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-16T23:46:03.053 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-16T23:46:03.054 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-16T23:46:03.054 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-16T23:46:03.054 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-16T23:46:03.054 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-16T23:46:03.054 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-16T23:46:03.055 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-16T23:46:03.055 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-16T23:46:03.055 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-16T23:46:03.055 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-16T23:46:03.055 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2023-10-16T23:46:03.056 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-16T23:46:03.056 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-16T23:46:03.056 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-16T23:46:03.056 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-16T23:46:03.056 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-16T23:46:03.057 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-16T23:46:03.058 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:03.058 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:03.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-base 2023-10-16T23:46:03.989 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:53 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:04.083 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-base-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:04.135 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:04.136 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:04.136 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:04.333 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install cephadm 2023-10-16T23:46:04.952 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:54 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:05.094 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:05.095 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:05.096 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:05.096 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 269 k 2023-10-16T23:46:05.096 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:05.097 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:05.097 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:05.097 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:05.097 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:05.097 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 269 k 2023-10-16T23:46:05.098 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 264 k 2023-10-16T23:46:05.098 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:05.447 INFO:teuthology.orchestra.run.smithi053.stdout:cephadm-18.0.0-6766.g06f14dfb.el9.noarch.rpm 766 kB/s | 269 kB 00:00 2023-10-16T23:46:05.448 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:05.449 INFO:teuthology.orchestra.run.smithi053.stdout:Total 762 kB/s | 269 kB 00:00 2023-10-16T23:46:05.449 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:05.454 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:05.455 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:05.457 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:05.458 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:05.534 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:05.954 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:06.038 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:06.461 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:06.790 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:06.790 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:06.790 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:06.790 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:06.791 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:06.791 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:06.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-immutable-object-cache 2023-10-16T23:46:07.613 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:01:57 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:07.757 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:07.758 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:07.758 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2023-10-16T23:46:07.758 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:07.758 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:07.759 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 144 k 2023-10-16T23:46:07.759 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:07.759 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:07.759 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:07.759 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:07.760 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:07.760 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 144 k 2023-10-16T23:46:07.760 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 438 k 2023-10-16T23:46:07.760 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:07.969 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-immutable-object-cache-18.0.0-6766.g06f14d 686 kB/s | 144 kB 00:00 2023-10-16T23:46:07.970 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:07.970 INFO:teuthology.orchestra.run.smithi053.stdout:Total 682 kB/s | 144 kB 00:00 2023-10-16T23:46:07.970 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:07.978 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:07.978 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:08.032 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:08.033 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:08.187 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:08.253 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6766.g06f14dfb. 1/1 2023-10-16T23:46:08.279 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6766.g06f14dfb. 1/1 2023-10-16T23:46:08.280 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:46:08.280 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-16T23:46:08.280 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:09.293 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6766.g06f14dfb. 1/1 2023-10-16T23:46:09.293 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:09.294 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:09.294 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:09.294 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:09.294 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:09.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr 2023-10-16T23:46:10.157 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:00 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:10.249 INFO:teuthology.orchestra.run.smithi053.stdout:Package ceph-mgr-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:10.302 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:10.303 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:10.303 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:10.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-dashboard 2023-10-16T23:46:10.996 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:00 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:11.144 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:11.145 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:11.145 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:11.145 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:11.145 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:11.146 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 1.8 M 2023-10-16T23:46:11.146 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:46:11.146 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 23 k 2023-10-16T23:46:11.146 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 15 k 2023-10-16T23:46:11.146 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-16T23:46:11.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-16T23:46:11.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-16T23:46:11.147 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2023-10-16T23:46:11.147 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-16T23:46:11.147 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:11.148 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:11.148 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:11.148 INFO:teuthology.orchestra.run.smithi053.stdout:Install 7 Packages 2023-10-16T23:46:11.148 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:11.149 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.1 M 2023-10-16T23:46:11.149 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 8.6 M 2023-10-16T23:46:11.149 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:11.731 INFO:teuthology.orchestra.run.smithi053.stdout:(1/7): ceph-prometheus-alerts-18.0.0-6766.g06f1 75 kB/s | 15 kB 00:00 2023-10-16T23:46:11.773 INFO:teuthology.orchestra.run.smithi053.stdout:(2/7): ceph-grafana-dashboards-18.0.0-6766.g06f 96 kB/s | 23 kB 00:00 2023-10-16T23:46:11.814 INFO:teuthology.orchestra.run.smithi053.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 108 kB/s | 9.0 kB 00:00 2023-10-16T23:46:11.881 INFO:teuthology.orchestra.run.smithi053.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 463 kB/s | 31 kB 00:00 2023-10-16T23:46:11.907 INFO:teuthology.orchestra.run.smithi053.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 306 kB/s | 41 kB 00:00 2023-10-16T23:46:11.949 INFO:teuthology.orchestra.run.smithi053.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6766.g06f14dfb 4.2 MB/s | 1.8 MB 00:00 2023-10-16T23:46:11.974 INFO:teuthology.orchestra.run.smithi053.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2023-10-16T23:46:12.025 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:12.025 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-10-16T23:46:12.026 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:12.026 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:12.026 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:12.055 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:12.058 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:12.360 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:12.442 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-16T23:46:12.476 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-16T23:46:12.532 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-16T23:46:12.562 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-16T23:46:12.591 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6766.g06f14dfb.el9.n 5/7 2023-10-16T23:46:12.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6766.g06f14dfb.el9. 6/7 2023-10-16T23:46:12.878 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6766.g06f14dfb.el9.noarc 7/7 2023-10-16T23:46:13.614 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6766.g06f14dfb.el9.noarc 7/7 2023-10-16T23:46:13.614 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6766.g06f14dfb.el9. 1/7 2023-10-16T23:46:13.615 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6766.g06f14dfb.el9.noarc 2/7 2023-10-16T23:46:13.615 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6766.g06f14dfb.el9.n 3/7 2023-10-16T23:46:13.615 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-16T23:46:13.615 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-16T23:46:13.615 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-16T23:46:13.950 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-16T23:46:13.950 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:13.951 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:13.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:13.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:13.951 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:13.951 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-16T23:46:13.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-16T23:46:13.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-16T23:46:13.952 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-16T23:46:13.952 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:13.952 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:14.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-16T23:46:14.930 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:04 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:15.078 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:15.079 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2023-10-16T23:46:15.079 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:15.080 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2023-10-16T23:46:15.080 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 7.4 M 2023-10-16T23:46:15.080 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:46:15.080 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-16T23:46:15.081 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-16T23:46:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-16T23:46:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-16T23:46:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-16T23:46:15.082 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-16T23:46:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:15.083 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:15.083 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2023-10-16T23:46:15.083 INFO:teuthology.orchestra.run.smithi053.stdout:Install 12 Packages 2023-10-16T23:46:15.083 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:15.084 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 37 M 2023-10-16T23:46:15.084 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 209 M 2023-10-16T23:46:15.084 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:15.222 INFO:teuthology.orchestra.run.smithi053.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2023-10-16T23:46:15.256 INFO:teuthology.orchestra.run.smithi053.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 525 kB/s | 17 kB 00:00 2023-10-16T23:46:15.339 INFO:teuthology.orchestra.run.smithi053.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-16T23:46:15.364 INFO:teuthology.orchestra.run.smithi053.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 320 kB/s | 34 kB 00:00 2023-10-16T23:46:15.431 INFO:teuthology.orchestra.run.smithi053.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.0 MB/s | 205 kB 00:00 2023-10-16T23:46:15.682 INFO:teuthology.orchestra.run.smithi053.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-10-16T23:46:15.782 INFO:teuthology.orchestra.run.smithi053.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2023-10-16T23:46:15.807 INFO:teuthology.orchestra.run.smithi053.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2023-10-16T23:46:15.908 INFO:teuthology.orchestra.run.smithi053.stdout:(9/12): ceph-mgr-diskprediction-local-18.0.0-67 8.9 MB/s | 7.4 MB 00:00 2023-10-16T23:46:16.116 INFO:teuthology.orchestra.run.smithi053.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2023-10-16T23:46:16.275 INFO:teuthology.orchestra.run.smithi053.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2023-10-16T23:46:16.300 INFO:teuthology.orchestra.run.smithi053.stdout:(12/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 501 kB/s | 195 kB 00:00 2023-10-16T23:46:16.302 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:16.303 INFO:teuthology.orchestra.run.smithi053.stdout:Total 30 MB/s | 37 MB 00:01 2023-10-16T23:46:16.303 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:16.326 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:16.326 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:16.897 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:16.897 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:17.637 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:17.723 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-16T23:46:17.754 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-16T23:46:17.851 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-16T23:46:17.900 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-16T23:46:18.094 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-16T23:46:18.157 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-16T23:46:18.236 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-16T23:46:19.064 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-16T23:46:19.180 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-16T23:46:20.501 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-16T23:46:21.569 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-16T23:46:21.614 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6766.g06f14 12/12 2023-10-16T23:46:24.139 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6766.g06f14 12/12 2023-10-16T23:46:24.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6766.g06f14 1/12 2023-10-16T23:46:24.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-16T23:46:24.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-16T23:46:24.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-16T23:46:24.140 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-16T23:46:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-16T23:46:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-16T23:46:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-16T23:46:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-16T23:46:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-16T23:46:24.142 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-16T23:46:24.624 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-16T23:46:24.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:24.625 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:24.625 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:24.625 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-16T23:46:24.626 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-16T23:46:24.626 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-16T23:46:24.626 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-16T23:46:24.626 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-16T23:46:24.626 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-16T23:46:24.627 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-16T23:46:24.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-16T23:46:24.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-16T23:46:24.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-16T23:46:24.627 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-16T23:46:24.628 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:24.628 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:24.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-rook 2023-10-16T23:46:25.608 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:15 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:25.754 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:25.756 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:25.756 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:25.756 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:25.756 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 47 k 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-16T23:46:25.757 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-16T23:46:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-16T23:46:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-16T23:46:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-16T23:46:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-16T23:46:25.758 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-16T23:46:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-16T23:46:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-16T23:46:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-16T23:46:25.759 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:25.759 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:25.760 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:25.760 INFO:teuthology.orchestra.run.smithi053.stdout:Install 13 Packages 2023-10-16T23:46:25.760 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:25.761 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 2.0 M 2023-10-16T23:46:25.761 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 26 M 2023-10-16T23:46:25.761 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:26.048 INFO:teuthology.orchestra.run.smithi053.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 190 kB/s | 24 kB 00:00 2023-10-16T23:46:26.073 INFO:teuthology.orchestra.run.smithi053.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2023-10-16T23:46:26.124 INFO:teuthology.orchestra.run.smithi053.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2023-10-16T23:46:26.149 INFO:teuthology.orchestra.run.smithi053.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2023-10-16T23:46:26.174 INFO:teuthology.orchestra.run.smithi053.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-10-16T23:46:26.199 INFO:teuthology.orchestra.run.smithi053.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2023-10-16T23:46:26.367 INFO:teuthology.orchestra.run.smithi053.stdout:(7/13): ceph-mgr-rook-18.0.0-6766.g06f14dfb.el9 106 kB/s | 47 kB 00:00 2023-10-16T23:46:26.392 INFO:teuthology.orchestra.run.smithi053.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 148 kB/s | 32 kB 00:00 2023-10-16T23:46:26.425 INFO:teuthology.orchestra.run.smithi053.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 66 kB/s | 15 kB 00:00 2023-10-16T23:46:26.576 INFO:teuthology.orchestra.run.smithi053.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 470 kB/s | 59 kB 00:00 2023-10-16T23:46:26.726 INFO:teuthology.orchestra.run.smithi053.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 599 kB/s | 90 kB 00:00 2023-10-16T23:46:26.784 INFO:teuthology.orchestra.run.smithi053.stdout:(12/13): python3-google-auth-2.23.2-1.el9.noarc 471 kB/s | 196 kB 00:00 2023-10-16T23:46:26.951 INFO:teuthology.orchestra.run.smithi053.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2023-10-16T23:46:26.955 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:26.956 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2023-10-16T23:46:26.956 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:26.970 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:26.971 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:27.076 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:27.077 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:27.611 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:27.780 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-16T23:46:27.837 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-16T23:46:27.901 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-16T23:46:27.937 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-16T23:46:27.979 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-16T23:46:28.075 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-16T23:46:28.186 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 7/13 2023-10-16T23:46:28.248 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-16T23:46:29.015 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-16T23:46:29.060 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-16T23:46:29.094 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-16T23:46:29.141 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-16T23:46:29.164 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:18.0.0-6766.g06f14dfb.el9.noarch 13/13 2023-10-16T23:46:30.012 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6766.g06f14dfb.el9.noarch 13/13 2023-10-16T23:46:30.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6766.g06f14dfb.el9.noarch 1/13 2023-10-16T23:46:30.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-16T23:46:30.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-16T23:46:30.013 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-16T23:46:30.014 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-16T23:46:30.014 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-16T23:46:30.014 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-16T23:46:30.014 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-16T23:46:30.014 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-16T23:46:30.015 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 10/13 2023-10-16T23:46:30.015 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-16T23:46:30.015 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-16T23:46:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-16T23:46:30.384 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:30.384 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-16T23:46:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-16T23:46:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-16T23:46:30.385 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-16T23:46:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-16T23:46:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-16T23:46:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-16T23:46:30.386 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-16T23:46:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-16T23:46:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-16T23:46:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-16T23:46:30.387 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-16T23:46:30.388 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:30.388 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:30.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-mgr-cephadm 2023-10-16T23:46:31.281 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:21 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:31.428 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:31.429 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:31.429 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:31.429 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:31.429 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:31.430 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 132 k 2023-10-16T23:46:31.430 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:46:31.430 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-16T23:46:31.430 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-16T23:46:31.430 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-16T23:46:31.431 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-16T23:46:31.431 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-16T23:46:31.431 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:31.431 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:31.431 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================== 2023-10-16T23:46:31.432 INFO:teuthology.orchestra.run.smithi053.stdout:Install 6 Packages 2023-10-16T23:46:31.432 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:31.432 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 6.8 M 2023-10-16T23:46:31.433 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 33 M 2023-10-16T23:46:31.433 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:31.786 INFO:teuthology.orchestra.run.smithi053.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-10-16T23:46:31.953 INFO:teuthology.orchestra.run.smithi053.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2023-10-16T23:46:31.978 INFO:teuthology.orchestra.run.smithi053.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6766.g06f14dfb.e 373 kB/s | 132 kB 00:00 2023-10-16T23:46:32.037 INFO:teuthology.orchestra.run.smithi053.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.1 MB/s | 548 kB 00:00 2023-10-16T23:46:32.062 INFO:teuthology.orchestra.run.smithi053.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 532 kB/s | 58 kB 00:00 2023-10-16T23:46:32.079 INFO:teuthology.orchestra.run.smithi053.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2023-10-16T23:46:32.082 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:32.082 INFO:teuthology.orchestra.run.smithi053.stdout:Total 10 MB/s | 6.8 MB 00:00 2023-10-16T23:46:32.083 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:32.092 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:32.092 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:32.187 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:32.187 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:32.487 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:32.669 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-16T23:46:32.723 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-16T23:46:33.273 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-16T23:46:33.369 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-16T23:46:33.437 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-16T23:46:33.468 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 6/6 2023-10-16T23:46:34.320 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 6/6 2023-10-16T23:46:34.320 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 1/6 2023-10-16T23:46:34.321 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-16T23:46:34.321 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-16T23:46:34.321 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-16T23:46:34.321 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-16T23:46:34.640 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-16T23:46:34.641 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:34.641 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:34.641 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:34.641 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-16T23:46:34.642 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-16T23:46:34.642 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-16T23:46:34.642 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-16T23:46:34.642 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-16T23:46:34.643 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:34.643 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:34.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-fuse 2023-10-16T23:46:35.548 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:25 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:35.693 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:35.694 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:35.694 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:35.695 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:35.695 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 831 k 2023-10-16T23:46:35.695 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2023-10-16T23:46:35.695 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-16T23:46:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:35.696 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:35.696 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:35.696 INFO:teuthology.orchestra.run.smithi053.stdout:Install 2 Packages 2023-10-16T23:46:35.696 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:35.697 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 910 k 2023-10-16T23:46:35.697 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 3.0 M 2023-10-16T23:46:35.697 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:35.848 INFO:teuthology.orchestra.run.smithi053.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 516 kB/s | 79 kB 00:00 2023-10-16T23:46:36.024 INFO:teuthology.orchestra.run.smithi053.stdout:(2/2): ceph-fuse-18.0.0-6766.g06f14dfb.el9.x86_ 2.5 MB/s | 831 kB 00:00 2023-10-16T23:46:36.025 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:36.025 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.7 MB/s | 910 kB 00:00 2023-10-16T23:46:36.025 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:36.046 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:36.047 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:36.126 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:36.126 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:36.310 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:36.444 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-16T23:46:36.508 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/2 2023-10-16T23:46:37.201 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 2/2 2023-10-16T23:46:37.202 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/2 2023-10-16T23:46:37.512 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-16T23:46:37.512 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:37.512 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:37.512 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-16T23:46:37.513 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:37.513 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:37.763 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-volume 2023-10-16T23:46:38.390 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:28 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:38.534 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:38.535 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:38.535 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2023-10-16T23:46:38.536 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:38.536 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:38.536 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:18.0.0-6766.g06f14dfb.el9 ceph-noarch 262 k 2023-10-16T23:46:38.536 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:38.536 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:38.537 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:38.537 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:38.537 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:38.537 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 262 k 2023-10-16T23:46:38.537 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 1.2 M 2023-10-16T23:46:38.538 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:38.761 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-volume-18.0.0-6766.g06f14dfb.el9.noarch.rp 1.1 MB/s | 262 kB 00:00 2023-10-16T23:46:38.762 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:38.762 INFO:teuthology.orchestra.run.smithi053.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-10-16T23:46:38.762 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:38.770 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:38.771 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:38.804 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:38.804 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:39.032 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:39.105 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:39.121 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:39.121 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:46:39.121 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-16T23:46:39.121 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:40.214 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:18.0.0-6766.g06f14dfb.el9.noarch 1/1 2023-10-16T23:46:40.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:40.215 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:40.215 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:18.0.0-6766.g06f14dfb.el9.noarch 2023-10-16T23:46:40.215 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:40.215 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:40.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados-devel 2023-10-16T23:46:41.150 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:31 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:41.296 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:41.297 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:41.297 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2023-10-16T23:46:41.298 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:41.298 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:41.298 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 127 k 2023-10-16T23:46:41.298 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:41.299 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:41.299 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:41.299 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:41.299 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:41.299 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 127 k 2023-10-16T23:46:41.300 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 460 k 2023-10-16T23:46:41.300 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:41.666 INFO:teuthology.orchestra.run.smithi053.stdout:librados-devel-18.0.0-6766.g06f14dfb.el9.x86_64 344 kB/s | 127 kB 00:00 2023-10-16T23:46:41.667 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:41.667 INFO:teuthology.orchestra.run.smithi053.stdout:Total 343 kB/s | 127 kB 00:00 2023-10-16T23:46:41.667 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:41.673 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:41.673 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:41.732 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:41.732 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:41.907 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:42.009 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:42.478 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:42.744 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:42.745 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:42.745 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:42.745 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:42.745 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:42.745 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:43.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs2 2023-10-16T23:46:43.633 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:33 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:43.726 INFO:teuthology.orchestra.run.smithi053.stdout:Package libcephfs2-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:43.779 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:43.780 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:43.781 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:43.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install libcephfs-devel 2023-10-16T23:46:44.473 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:34 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:44.619 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:44.620 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:44.620 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2023-10-16T23:46:44.621 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:44.621 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:44.621 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 31 k 2023-10-16T23:46:44.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:44.622 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:44.622 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:44.623 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:44.623 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:44.623 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 31 k 2023-10-16T23:46:44.624 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 139 k 2023-10-16T23:46:44.624 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:44.778 INFO:teuthology.orchestra.run.smithi053.stdout:libcephfs-devel-18.0.0-6766.g06f14dfb.el9.x86_6 199 kB/s | 31 kB 00:00 2023-10-16T23:46:44.778 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:44.779 INFO:teuthology.orchestra.run.smithi053.stdout:Total 197 kB/s | 31 kB 00:00 2023-10-16T23:46:44.779 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:44.782 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:44.782 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:44.812 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:44.812 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:44.909 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:45.020 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:45.448 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:45.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:45.803 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:45.803 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:45.803 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:45.804 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:45.804 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:46.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librados2 2023-10-16T23:46:46.724 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:36 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:46.817 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:46.869 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:46.870 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:46.870 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:46.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install librbd1 2023-10-16T23:46:47.558 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:37 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:47.651 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:47.703 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:47.704 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:47.705 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:47.777 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rados 2023-10-16T23:46:48.391 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:38 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:48.483 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rados-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:48.536 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:48.538 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:48.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rgw 2023-10-16T23:46:49.228 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:39 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:49.321 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rgw-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:49.374 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:49.375 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:49.375 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:49.442 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-cephfs 2023-10-16T23:46:50.064 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:40 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:50.157 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-cephfs-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:50.211 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:50.212 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:50.212 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:50.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install python3-rbd 2023-10-16T23:46:50.913 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:40 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:51.005 INFO:teuthology.orchestra.run.smithi053.stdout:Package python3-rbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 is already installed. 2023-10-16T23:46:51.058 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:51.059 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2023-10-16T23:46:51.059 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:51.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-fuse 2023-10-16T23:46:51.749 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:41 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:51.893 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:51.894 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:51.894 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2023-10-16T23:46:51.895 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:51.895 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:51.895 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 87 k 2023-10-16T23:46:51.895 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:51.896 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:51.896 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:51.896 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:51.896 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:51.896 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 87 k 2023-10-16T23:46:51.897 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 252 k 2023-10-16T23:46:51.897 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:52.131 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-fuse-18.0.0-6766.g06f14dfb.el9.x86_64.rpm 368 kB/s | 87 kB 00:00 2023-10-16T23:46:52.132 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:52.132 INFO:teuthology.orchestra.run.smithi053.stdout:Total 365 kB/s | 87 kB 00:00 2023-10-16T23:46:52.132 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:52.139 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:52.139 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:52.197 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:52.198 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:52.339 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:52.433 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:52.870 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:53.217 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:53.218 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:53.218 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:53.218 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:53.218 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:53.219 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:53.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-mirror 2023-10-16T23:46:54.063 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:44 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:54.210 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:54.210 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:54.211 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repo Size 2023-10-16T23:46:54.211 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:54.211 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:54.212 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 3.0 M 2023-10-16T23:46:54.212 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:54.212 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:54.212 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:54.213 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:54.213 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:54.213 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 3.0 M 2023-10-16T23:46:54.214 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 13 M 2023-10-16T23:46:54.214 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:55.334 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-mirror-18.0.0-6766.g06f14dfb.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2023-10-16T23:46:55.335 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:55.335 INFO:teuthology.orchestra.run.smithi053.stdout:Total 2.7 MB/s | 3.0 MB 00:01 2023-10-16T23:46:55.336 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:55.347 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:55.347 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:55.426 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:55.427 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:55.889 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:55.943 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:55.967 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:55.968 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-16T23:46:55.968 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-16T23:46:55.968 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T23:46:55.968 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-16T23:46:55.968 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:57.176 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:57.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:57.177 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:57.177 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:57.177 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:57.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install rbd-nbd 2023-10-16T23:46:58.041 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:02:48 ago on Mon 16 Oct 2023 11:44:10 PM UTC. 2023-10-16T23:46:58.187 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2023-10-16T23:46:58.188 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2023-10-16T23:46:58.188 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:58.188 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2023-10-16T23:46:58.188 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:18.0.0-6766.g06f14dfb.el9 ceph 168 k 2023-10-16T23:46:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:58.189 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2023-10-16T23:46:58.189 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2023-10-16T23:46:58.189 INFO:teuthology.orchestra.run.smithi053.stdout:Install 1 Package 2023-10-16T23:46:58.189 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:58.190 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 168 k 2023-10-16T23:46:58.190 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 533 k 2023-10-16T23:46:58.190 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2023-10-16T23:46:58.404 INFO:teuthology.orchestra.run.smithi053.stdout:rbd-nbd-18.0.0-6766.g06f14dfb.el9.x86_64.rpm 781 kB/s | 168 kB 00:00 2023-10-16T23:46:58.405 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2023-10-16T23:46:58.405 INFO:teuthology.orchestra.run.smithi053.stdout:Total 776 kB/s | 168 kB 00:00 2023-10-16T23:46:58.405 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2023-10-16T23:46:58.413 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2023-10-16T23:46:58.413 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2023-10-16T23:46:58.472 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2023-10-16T23:46:58.473 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2023-10-16T23:46:58.622 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2023-10-16T23:46:58.731 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:59.501 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 1/1 2023-10-16T23:46:59.502 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:59.502 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2023-10-16T23:46:59.502 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:18.0.0-6766.g06f14dfb.el9.x86_64 2023-10-16T23:46:59.502 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:46:59.503 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2023-10-16T23:46:59.749 DEBUG:teuthology.parallel:result is None 2023-10-16T23:46:59.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=06f14dfbeeb0dce341775222d82ea9644e37c496 2023-10-16T23:46:59.893 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-16T23:46:59.914 INFO:teuthology.orchestra.run.smithi053.stdout:18.0.0-6766.g06f14dfb.el9 2023-10-16T23:46:59.914 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6766.g06f14dfb.el9 2023-10-16T23:46:59.915 INFO:teuthology.task.install:The correct ceph version 18.0.0-6766.g06f14dfb is installed. 2023-10-16T23:46:59.915 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-16T23:46:59.916 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:46:59.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-16T23:46:59.986 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-16T23:46:59.987 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:46:59.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2023-10-16T23:47:00.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-16T23:47:00.127 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-16T23:47:00.127 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:00.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-16T23:47:00.197 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-16T23:47:00.268 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-16T23:47:00.269 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:00.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2023-10-16T23:47:00.338 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-16T23:47:00.408 INFO:teuthology.run_tasks:Running task ceph... 2023-10-16T23:47:00.473 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-16T23:47:00.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2023-10-16T23:47:00.503 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-16T23:47:00.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-16T23:47:00.574 INFO:tasks.ceph:Creating extra log directories... 2023-10-16T23:47:00.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-16T23:47:00.655 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-16T23:47:00.655 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-16T23:47:00.655 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682', 'branch': 'wip-crimson-scrub-testing-2023-10-15', 'description': 'crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7430682', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-crimson-scrub-testing-2023-10-15'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496'}}, 'workunit': {'branch': 'wip-crimson-scrub-testing-2023-10-15', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496'}}, 'owner': 'scheduled_sjust@teuthology', 'priority': 51, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 4255, 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-crimson-scrub-testing-2023-10-15', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496', 'targets': {'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeOtfSQGEi39/FCzuey/qkBwXw9MuQMnDkE0Bc7f2K1cUAKQ8bY00fJA8vzUJfVlCG/fRGvBKEk5YDeH430UO4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '06f14dfbeeb0dce341775222d82ea9644e37c496', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-16_23:13:53', 'tube': 'smithi', 'user': 'sjust', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-16T23:47:00.656 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-16T23:47:00.700 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2023-10-16T23:47:00.769 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:00.769 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2023-10-16T23:47:00.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-16T23:47:00.825 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2023-10-16T23:47:00.879 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-16T23:47:00.880 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T23:47:00.880 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-10-16T23:47:00.880 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T23:47:00.880 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T23:47:00.881 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-10-16 23:46:56.225529615 +0000 2023-10-16T23:47:00.881 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-10-16 23:42:51.387626017 +0000 2023-10-16T23:47:00.881 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-10-16 23:42:51.387626017 +0000 2023-10-16T23:47:00.881 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2023-10-16 23:42:51.387626017 +0000 2023-10-16T23:47:00.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-16T23:47:00.948 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-10-16T23:47:00.949 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-10-16T23:47:00.949 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.1728e-05 s, 5.6 MB/s 2023-10-16T23:47:00.950 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-16T23:47:01.006 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2023-10-16T23:47:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-16T23:47:01.061 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T23:47:01.061 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-10-16T23:47:01.062 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T23:47:01.062 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T23:47:01.062 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-10-16 23:46:56.226529596 +0000 2023-10-16T23:47:01.062 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-10-16 23:42:51.890616398 +0000 2023-10-16T23:47:01.062 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-10-16 23:42:51.890616398 +0000 2023-10-16T23:47:01.063 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2023-10-16 23:42:51.890616398 +0000 2023-10-16T23:47:01.063 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-16T23:47:01.129 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-10-16T23:47:01.130 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-10-16T23:47:01.130 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.3628e-05 s, 5.5 MB/s 2023-10-16T23:47:01.131 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-16T23:47:01.187 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2023-10-16T23:47:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-16T23:47:01.242 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T23:47:01.242 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 700 Links: 1 2023-10-16T23:47:01.242 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T23:47:01.243 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T23:47:01.243 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-10-16 23:46:56.226529596 +0000 2023-10-16T23:47:01.243 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-10-16 23:42:52.369607239 +0000 2023-10-16T23:47:01.243 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-10-16 23:42:52.369607239 +0000 2023-10-16T23:47:01.243 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2023-10-16 23:42:52.369607239 +0000 2023-10-16T23:47:01.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-16T23:47:01.311 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-10-16T23:47:01.311 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-10-16T23:47:01.311 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.6639e-05 s, 5.3 MB/s 2023-10-16T23:47:01.312 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-16T23:47:01.368 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2023-10-16T23:47:01.423 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-16T23:47:01.423 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-16T23:47:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-10-16T23:47:01.423 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-16T23:47:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2023-10-16T23:47:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2023-10-16 23:46:56.226529596 +0000 2023-10-16T23:47:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2023-10-16 23:42:52.857597907 +0000 2023-10-16T23:47:01.424 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2023-10-16 23:42:52.857597907 +0000 2023-10-16T23:47:01.424 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2023-10-16 23:42:52.857597907 +0000 2023-10-16T23:47:01.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-16T23:47:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2023-10-16T23:47:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2023-10-16T23:47:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 9.409e-05 s, 5.4 MB/s 2023-10-16T23:47:01.493 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-16T23:47:01.549 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-16T23:47:01.549 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-16T23:47:01.549 INFO:tasks.ceph:Generating config... 2023-10-16T23:47:01.551 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-10-16T23:47:01.551 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-10-16T23:47:01.551 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-10-16T23:47:01.551 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-10-16T23:47:01.552 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-10-16T23:47:01.552 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-16T23:47:01.552 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-16T23:47:01.552 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-16T23:47:01.553 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-16T23:47:01.553 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-16T23:47:01.553 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-16T23:47:01.553 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-16T23:47:01.554 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-16T23:47:01.554 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-16T23:47:01.554 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-16T23:47:01.554 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-16T23:47:01.555 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-16T23:47:01.555 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-16T23:47:01.555 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-16T23:47:01.556 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-16T23:47:01.556 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-16T23:47:01.556 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-16T23:47:01.556 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-16T23:47:01.557 INFO:tasks.ceph:Setting up mon.a... 2023-10-16T23:47:01.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-16T23:47:01.650 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.keyring 2023-10-16T23:47:01.653 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-16T23:47:01.708 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T23:47:01.738 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.53')] 2023-10-16T23:47:01.739 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.53', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson seastar smp': 3, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-16T23:47:01.740 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:01.740 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T23:47:01.793 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.53 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:01.878 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:01.877+0000 7efd4608ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:01.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:01.877+0000 7efd4608ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:01.879 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:01.877+0000 7efd4608ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:01.880 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:01.880 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: generated fsid ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:01.881 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = pacific 2023-10-16T23:47:01.881 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 0 2023-10-16T23:47:01.881 INFO:teuthology.orchestra.run.smithi053.stdout:fsid ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:01.881 INFO:teuthology.orchestra.run.smithi053.stdout:last_changed 2023-10-16T23:47:01.878544+0000 2023-10-16T23:47:01.881 INFO:teuthology.orchestra.run.smithi053.stdout:created 2023-10-16T23:47:01.878544+0000 2023-10-16T23:47:01.882 INFO:teuthology.orchestra.run.smithi053.stdout:min_mon_release 16 (pacific) 2023-10-16T23:47:01.882 INFO:teuthology.orchestra.run.smithi053.stdout:election_strategy: 1 2023-10-16T23:47:01.882 INFO:teuthology.orchestra.run.smithi053.stdout:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2023-10-16T23:47:01.882 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-16T23:47:01.883 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-16T23:47:01.898 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ec8ab2be-8061-4c8a-9ea6-1163eadae66a... 2023-10-16T23:47:01.899 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-16T23:47:01.998 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2023-10-16T23:47:01.998 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2023-10-16T23:47:01.998 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-16T23:47:01.999 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2023-10-16T23:47:01.999 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:01.999 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2023-10-16T23:47:01.999 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:01.999 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2023-10-16T23:47:02.000 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.000 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2023-10-16T23:47:02.000 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2023-10-16T23:47:02.000 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.000 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2023-10-16T23:47:02.001 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2023-10-16T23:47:02.001 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.001 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-16T23:47:02.001 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2023-10-16T23:47:02.001 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.002 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2023-10-16T23:47:02.003 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2023-10-16T23:47:02.003 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2023-10-16T23:47:02.003 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2023-10-16T23:47:02.003 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2023-10-16T23:47:02.003 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2023-10-16T23:47:02.004 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2023-10-16T23:47:02.004 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-16T23:47:02.004 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-16T23:47:02.004 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2023-10-16T23:47:02.004 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.005 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-16T23:47:02.005 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.005 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2023-10-16T23:47:02.005 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.005 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2023-10-16T23:47:02.006 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.006 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2023-10-16T23:47:02.006 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2023-10-16T23:47:02.006 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2023-10-16T23:47:02.006 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = 172.21.15.53 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: ms client mode = crc 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: ms cluster mode = crc 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: ms mon client mode = crc 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: ms mon cluster mode = crc 2023-10-16T23:47:02.007 INFO:teuthology.orchestra.run.smithi053.stdout: ms mon service mode = crc 2023-10-16T23:47:02.008 INFO:teuthology.orchestra.run.smithi053.stdout: ms service mode = crc 2023-10-16T23:47:02.008 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:02.008 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.008 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2023-10-16T23:47:02.008 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2023-10-16T23:47:02.009 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.009 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2023-10-16T23:47:02.009 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2023-10-16T23:47:02.009 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2023-10-16T23:47:02.009 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2023-10-16T23:47:02.010 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2023-10-16T23:47:02.011 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.011 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2023-10-16T23:47:02.011 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2023-10-16T23:47:02.011 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.011 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2023-10-16T23:47:02.012 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.012 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2023-10-16T23:47:02.012 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.012 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2023-10-16T23:47:02.012 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2023-10-16T23:47:02.013 INFO:teuthology.orchestra.run.smithi053.stdout: debug monc = 20 2023-10-16T23:47:02.014 INFO:teuthology.orchestra.run.smithi053.stdout: crimson seastar smp = 3 2023-10-16T23:47:02.014 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-10-16T23:47:02.014 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2023-10-16T23:47:02.014 INFO:teuthology.orchestra.run.smithi053.stdout: osd shutdown pgref assert = True 2023-10-16T23:47:02.014 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2023-10-16T23:47:02.015 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2023-10-16T23:47:02.016 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2023-10-16T23:47:02.016 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2023-10-16T23:47:02.016 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.016 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2023-10-16T23:47:02.016 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2023-10-16T23:47:02.017 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2023-10-16T23:47:02.017 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2023-10-16T23:47:02.017 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2023-10-16T23:47:02.017 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2023-10-16T23:47:02.017 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.018 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-16T23:47:02.019 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2023-10-16T23:47:02.019 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2023-10-16T23:47:02.019 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.019 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2023-10-16T23:47:02.019 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-16T23:47:02.020 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-16T23:47:02.020 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.020 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2023-10-16T23:47:02.020 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2023-10-16T23:47:02.021 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.021 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2023-10-16T23:47:02.021 INFO:teuthology.orchestra.run.smithi053.stdout: mon min osdmap epochs = 50 2023-10-16T23:47:02.021 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune interval = 2 2023-10-16T23:47:02.021 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune min = 15 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout: mon osdmap full prune txsize = 2 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout: paxos service trim min = 10 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default crimson = True 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2023-10-16T23:47:02.022 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2023-10-16T23:47:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2023-10-16T23:47:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2023-10-16T23:47:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-16T23:47:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-16T23:47:02.023 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2023-10-16T23:47:02.024 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-16T23:47:02.025 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-16T23:47:02.130 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-16T23:47:02.131 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:02.131 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-16T23:47:02.146 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:02.147 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-16T23:47:02.200 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2023-10-16T23:47:02.201 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:02.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-16T23:47:02.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-16T23:47:02.286 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:02.286 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:02.341 INFO:tasks.ceph:Setting up mon nodes... 2023-10-16T23:47:02.341 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-16T23:47:02.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T23:47:02.448 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-16T23:47:02.451 INFO:tasks.ceph:Setting up mds nodes... 2023-10-16T23:47:02.451 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-16T23:47:02.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-16T23:47:02.503 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-16T23:47:02.522 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-16T23:47:02.523 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-16T23:47:02.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-16T23:47:02.553 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-16T23:47:02.554 INFO:tasks.ceph:role: osd.0 2023-10-16T23:47:02.554 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-16T23:47:02.554 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-16T23:47:02.623 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T23:47:02.624 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T23:47:02.624 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T23:47:02.624 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T23:47:02.624 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T23:47:02.625 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-10-16T23:47:02.625 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T23:47:02.625 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T23:47:02.625 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T23:47:02.625 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T23:47:02.655 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-10-16T23:47:02.657 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-10-16T23:47:02.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-16T23:47:02.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-16T23:47:02.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-16T23:47:02.879 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-16T23:47:02.879 INFO:tasks.ceph:role: osd.1 2023-10-16T23:47:02.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-16T23:47:02.880 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-16T23:47:02.949 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T23:47:02.949 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T23:47:02.950 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T23:47:02.950 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T23:47:02.950 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T23:47:02.950 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-10-16T23:47:02.950 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T23:47:02.951 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T23:47:02.951 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T23:47:02.951 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T23:47:02.981 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-10-16T23:47:02.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-10-16T23:47:02.983 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-16T23:47:03.058 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-16T23:47:03.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-16T23:47:03.203 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-16T23:47:03.203 INFO:tasks.ceph:role: osd.2 2023-10-16T23:47:03.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-16T23:47:03.204 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-16T23:47:03.274 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-16T23:47:03.274 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-16T23:47:03.274 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-16T23:47:03.274 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-16T23:47:03.275 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-16T23:47:03.306 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2023-10-16T23:47:03.309 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2023-10-16T23:47:03.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-16T23:47:03.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-16T23:47:03.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:04.045 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:04.046 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,045 seastar - Reactor backend: linux-aio 2023-10-16T23:47:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,073 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,073 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,073 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,073 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:04.076 INFO:teuthology.orchestra.run.smithi053.stderr:==79996==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:==79996==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd1aa27000; bottom 0x7f716f84a000; size: 0x008bab1dd000 (599871311872) 2023-10-16T23:47:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:04.083 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:04.119 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,118 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-16T23:47:04.119 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,118 [shard 0] osd - get_early_config: set --smp 3 2023-10-16T23:47:04.634 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:04.635 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:04.635 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:04.635 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:04.635 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:04.635 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:04.683 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:04.683 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,683 seastar - Reactor backend: linux-aio 2023-10-16T23:47:04.701 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,701 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:04.701 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,701 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:04.701 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,701 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:04.708 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:04,708 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:==79994==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:==79994==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd1aa27000; bottom 0x7f716d633000; size: 0x008bad3f4000 (599907057664) 2023-10-16T23:47:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:04.718 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d334470 0x2d33581e 0x2d336d6c 0x2d3370f0 0x2d33723a 0x2d337690 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f4ba4e 0x26f4e097 0x26f4fea8 0x26f5270d 0x26d67bf9 0x1f5540ba 0x1f4c8d83 0x1f4cccef 0x2d6ab037 2023-10-16T23:47:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:kernel callstack: 2023-10-16T23:47:08.352 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:08.352 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:08.353 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:08.353 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:08.353 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:08.353 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:08.353 INFO:teuthology.orchestra.run.smithi053.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:08.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-16T23:47:08.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:08.976 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:08,975 seastar - Reactor backend: linux-aio 2023-10-16T23:47:08.993 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:08,993 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:08.994 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:08,993 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:08.994 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:08,993 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:08.994 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:08,993 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:08.996 INFO:teuthology.orchestra.run.smithi053.stderr:==80805==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:09.002 INFO:teuthology.orchestra.run.smithi053.stderr:==80805==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcf1c59000; bottom 0x7f9a271ce000; size: 0x0062caa8b000 (424306847744) 2023-10-16T23:47:09.002 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:09.002 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d334470 0x2d33581e 0x2d336d6c 0x2d3370f0 0x2d33723a 0x2d337690 0x54daf 0x238e0fca 0x237b60ef 0x237db285 0x23686086 0x236b3ad9 0x25681d51 0x25687479 0x25689a14 0x2568abed 0x2324334d 0x23293492 0x232b7705 0x2d2d9483 0x2d36e90a 0x2d58b892 0x2d58d6dd 0x2d002994 0x2d004895 0x1f615b2c 0x1f61fc99 0x1f498f27 0x3feaf 0x3ff5f 0x1ef52a04 2023-10-16T23:47:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:kernel callstack: 2023-10-16T23:47:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,037 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-16T23:47:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,037 [shard 0] osd - get_early_config: set --smp 3 2023-10-16T23:47:09.548 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:09.549 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:09.549 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:09.549 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:09.549 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,612 seastar - Reactor backend: linux-aio 2023-10-16T23:47:09.630 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,630 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:09.630 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,630 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:09.630 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,630 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:09.637 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:09,637 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:09.640 INFO:teuthology.orchestra.run.smithi053.stderr:==80803==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:09.646 INFO:teuthology.orchestra.run.smithi053.stderr:==80803==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcf1c59000; bottom 0x7f9a24fb7000; size: 0x0062ccca2000 (424342593536) 2023-10-16T23:47:09.646 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:09.647 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:09.727 INFO:teuthology.orchestra.run.smithi053.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d334470 0x2d33581e 0x2d336d6c 0x2d3370f0 0x2d33723a 0x2d337690 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f4ba4e 0x26f4e097 0x26f4fea8 0x26f5270d 0x26d67bf9 0x1f5540ba 0x1f4c8d83 0x1f4cccef 0x2d6ab037 2023-10-16T23:47:09.728 INFO:teuthology.orchestra.run.smithi053.stderr:kernel callstack: 2023-10-16T23:47:13.252 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:13.253 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:13.253 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:13.253 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:13.253 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:13.254 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:13.254 INFO:teuthology.orchestra.run.smithi053.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:13.306 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-16T23:47:13.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:13.894 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:13.895 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,894 seastar - Reactor backend: linux-aio 2023-10-16T23:47:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,910 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,910 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,910 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,910 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:==81614==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:==81614==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1e563000; bottom 0x7f6283840000; size: 0x009b9ad23000 (668317396992) 2023-10-16T23:47:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:13.945 INFO:teuthology.orchestra.run.smithi053.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2d334470 0x2d33581e 0x2d336d6c 0x2d3370f0 0x2d33723a 0x2d337690 0x54daf 0x3985d 0x94437 0x94d46 0x237b0137 0x2394aa8b 0x2394ad22 0x237dcca1 0x23686086 0x236b3ad9 0x25681d51 0x25687479 0x25689a14 0x2568abed 0x2324334d 0x23293492 0x232b7705 0x2d2d9483 0x2d36e90a 0x2d58b892 0x2d58d6dd 0x2d002994 0x2d004895 0x1f615b2c 0x1f61fc99 0x1f498f27 0x3feaf 0x3ff5f 0x1ef52a04 2023-10-16T23:47:13.946 INFO:teuthology.orchestra.run.smithi053.stderr:kernel callstack: 2023-10-16T23:47:13.955 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,955 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2023-10-16T23:47:13.955 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:13,955 [shard 0] osd - get_early_config: set --smp 3 2023-10-16T23:47:14.469 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:14.469 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:14.469 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:14.469 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:14.469 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:14.470 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-16T23:47:14.527 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:14,527 seastar - Reactor backend: linux-aio 2023-10-16T23:47:14.545 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:14,545 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-16T23:47:14.545 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:14,545 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-16T23:47:14.546 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:14,545 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-16T23:47:14.553 INFO:teuthology.orchestra.run.smithi053.stderr:INFO 2023-10-16 23:47:14,552 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-16T23:47:14.555 INFO:teuthology.orchestra.run.smithi053.stderr:==81611==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-16T23:47:14.561 INFO:teuthology.orchestra.run.smithi053.stderr:==81611==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe1e563000; bottom 0x7f6281629000; size: 0x009b9cf3a000 (668353142784) 2023-10-16T23:47:14.562 INFO:teuthology.orchestra.run.smithi053.stderr:False positive error reports may follow 2023-10-16T23:47:14.562 INFO:teuthology.orchestra.run.smithi053.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-16T23:47:14.644 INFO:teuthology.orchestra.run.smithi053.stderr:Reactor stalled for 33 ms on shard 0. Backtrace: 0x45d5d 0x2d334470 0x2d33581e 0x2d336d6c 0x2d3370f0 0x2d33723a 0x2d337690 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26f4ba4e 0x26f4e097 0x26f4fea8 0x26f5270d 0x26d67bf9 0x1f5540ba 0x1f4c8d83 0x1f4cccef 0x2d6ab037 2023-10-16T23:47:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffb16681d2 0xffffffffb166866d 0xffffffffb0b4fa8a 0xffffffffb165918c 0xffffffffb18000aa 2023-10-16T23:47:18.176 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:18.177 INFO:teuthology.orchestra.run.smithi053.stderr:Suppressions used: 2023-10-16T23:47:18.177 INFO:teuthology.orchestra.run.smithi053.stderr: count bytes template 2023-10-16T23:47:18.177 INFO:teuthology.orchestra.run.smithi053.stderr: 1 8 MallocExtension::Register 2023-10-16T23:47:18.177 INFO:teuthology.orchestra.run.smithi053.stderr:----------------------------------------------------- 2023-10-16T23:47:18.178 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-16T23:47:18.178 INFO:teuthology.orchestra.run.smithi053.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid ec8ab2be-8061-4c8a-9ea6-1163eadae66a 2023-10-16T23:47:18.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-16T23:47:18.251 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-16T23:47:18.251 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:18.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-16T23:47:18.321 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:18.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-16T23:47:18.391 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:18.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-16T23:47:18.463 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:18.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-16T23:47:18.533 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:18.534 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-16T23:47:18.588 INFO:tasks.ceph:Adding keys to all mons... 2023-10-16T23:47:18.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-16T23:47:18.656 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2023-10-16T23:47:18.657 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQB2yy1lcFa4GhAApygfCd0OKGFseHn2T/qs2w== 2023-10-16T23:47:18.657 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2023-10-16T23:47:18.657 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQB4yy1lnTW7LRAAuKTzkxh6kwl3+95nLhxbGQ== 2023-10-16T23:47:18.657 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2023-10-16T23:47:18.658 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQB9yy1lYVdjKRAAkxUtmmezsrej5tKIN096rw== 2023-10-16T23:47:18.658 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2023-10-16T23:47:18.658 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCCyy1lClVuJBAAIx0+jmfob8PfB+kvxv9b0w== 2023-10-16T23:47:18.658 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2023-10-16T23:47:18.658 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQB2yy1l2hkCHhAAE60TXS+Cnwa9bWv1JGYOTA== 2023-10-16T23:47:18.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-16T23:47:18.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T23:47:18.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T23:47:18.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-16T23:47:18.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-16T23:47:19.012 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-16T23:47:19.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-16T23:47:19.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-16T23:47:19.147 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.146+0000 7fb2aa245980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.147 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.146+0000 7fb2aa245980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.162 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-16T23:47:19.193 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-16T23:47:19.248 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-16T23:47:19.249 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-16T23:47:19.249 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-16T23:47:19.290 INFO:tasks.ceph.mon.a:Started 2023-10-16T23:47:19.291 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-16T23:47:19.291 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-16T23:47:19.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-16T23:47:19.294 INFO:tasks.ceph.mgr.x:Started 2023-10-16T23:47:19.294 INFO:tasks.ceph:Setting crush tunables to default 2023-10-16T23:47:19.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd crush tunables default 2023-10-16T23:47:19.376 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:19.374+0000 7f6d09819f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.376 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:47:19.374+0000 7f8595e10980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.377 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:47:19.374+0000 7f8595e10980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.377 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:47:19.375+0000 7f8595e10980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.407 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.401+0000 7f283bb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.431 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.430+0000 7f283bb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.502 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2023-10-16T23:47:19.513 INFO:tasks.ceph:check_enable_crimson: True 2023-10-16T23:47:19.513 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-16T23:47:19.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-16T23:47:19.581 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:19.580+0000 7f6d09819f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.581 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:19.580+0000 7f6d09819f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.634 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.628+0000 7f5748e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.637 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:19.635+0000 7f5748e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:19.719 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:47:19.718+0000 7f858ddf1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-16T23:47:19.719 INFO:tasks.ceph.mon.a.smithi053.stderr:Please be aware that this feature is experimental, untested, 2023-10-16T23:47:19.720 INFO:tasks.ceph.mon.a.smithi053.stderr:unsupported, and may result in data corruption, data loss, 2023-10-16T23:47:19.720 INFO:tasks.ceph.mon.a.smithi053.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-16T23:47:19.720 INFO:tasks.ceph.mon.a.smithi053.stderr:feature with important data. 2023-10-16T23:47:19.720 INFO:tasks.ceph.mon.a.smithi053.stderr: 2023-10-16T23:47:20.079 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.078+0000 7f6d09819f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-16T23:47:20.133 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.131+0000 7f6d09819f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-16T23:47:20.518 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-16T23:47:20.519 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:20.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-16T23:47:20.551 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:20.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-16T23:47:20.621 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:47:20.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-16T23:47:20.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new cd6c98cf-26a2-4a1a-8d24-e07b286f63c2 0 2023-10-16T23:47:20.698 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.696+0000 7f6d09819f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-16T23:47:20.754 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.753+0000 7f6d09819f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-16T23:47:20.803 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.802+0000 7f6d09819f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-16T23:47:20.808 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:20.802+0000 7fa674c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:20.811 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:20.810+0000 7fa674c24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:20.861 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.859+0000 7f6d09819f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-16T23:47:20.892 INFO:teuthology.orchestra.run.smithi053.stdout:0 2023-10-16T23:47:20.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 6e2af387-5606-4858-bcdc-e57480518213 1 2023-10-16T23:47:20.906 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:20.905+0000 7f6d09819f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-16T23:47:21.013 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.006+0000 7fe6f5b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.014 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.013+0000 7f6d09819f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-16T23:47:21.016 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.015+0000 7fe6f5b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.088 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.087+0000 7f6d09819f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-16T23:47:21.096 INFO:teuthology.orchestra.run.smithi053.stdout:1 2023-10-16T23:47:21.107 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 34c131e5-7d98-4a4a-a569-6367d63d959f 2 2023-10-16T23:47:21.138 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.136+0000 7f6d09819f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-16T23:47:21.193 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.192+0000 7f6d09819f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-16T23:47:21.244 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.243+0000 7f6d09819f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-16T23:47:21.249 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.241+0000 7f3fb682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.250 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.249+0000 7f3fb682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.330 INFO:teuthology.orchestra.run.smithi053.stdout:2 2023-10-16T23:47:21.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-16T23:47:21.377 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-16T23:47:21.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-16T23:47:21.419 INFO:tasks.ceph.osd.0:Started 2023-10-16T23:47:21.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-16T23:47:21.456 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-16T23:47:21.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-16T23:47:21.498 INFO:tasks.ceph.osd.1:Started 2023-10-16T23:47:21.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-16T23:47:21.538 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-16T23:47:21.538 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-16T23:47:21.581 INFO:tasks.ceph.osd.2:Started 2023-10-16T23:47:21.581 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T23:47:21.688 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.687+0000 7f6d09819f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-16T23:47:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.708+0000 7ff2c76f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:21.717+0000 7ff2c76f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:21.790 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:21.791 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":6,"fsid":"ec8ab2be-8061-4c8a-9ea6-1163eadae66a","created":"2023-10-16T23:47:19.419081+0000","modified":"2023-10-16T23:47:21.329690+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T23:47:21.329690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"cd6c98cf-26a2-4a1a-8d24-e07b286f63c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e2af387-5606-4858-bcdc-e57480518213","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c131e5-7d98-4a4a-a569-6367d63d959f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T23:47:21.807 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-16T23:47:21.808 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-16T23:47:21.859 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.857+0000 7f6d09819f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-16T23:47:21.995 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:21.994+0000 7f6d09819f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-16T23:47:22.048 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.047+0000 7f6d09819f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-16T23:47:22.109 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T23:47:22.158 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.157+0000 7f6d09819f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-16T23:47:22.213 INFO:teuthology.misc.health.smithi053.stderr:2023-10-16T23:47:22.208+0000 7fb7dc86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:22.216 INFO:teuthology.misc.health.smithi053.stderr:2023-10-16T23:47:22.215+0000 7fb7dc86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:22.264 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.262+0000 7f6d09819f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-16T23:47:22.283 INFO:teuthology.misc.health.smithi053.stdout: 2023-10-16T23:47:22.283 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":6,"fsid":"ec8ab2be-8061-4c8a-9ea6-1163eadae66a","created":"2023-10-16T23:47:19.419081+0000","modified":"2023-10-16T23:47:21.329690+0000","last_up_change":"0.000000","last_in_change":"2023-10-16T23:47:21.329690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"cd6c98cf-26a2-4a1a-8d24-e07b286f63c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6e2af387-5606-4858-bcdc-e57480518213","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"34c131e5-7d98-4a4a-a569-6367d63d959f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T23:47:22.294 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-16T23:47:22.419 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.418+0000 7f6d09819f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-16T23:47:22.466 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.465+0000 7f6d09819f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-16T23:47:22.621 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:22.620+0000 7f6d09819f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-16T23:47:23.459 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:23.457+0000 7f6d09819f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-16T23:47:23.758 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:23.756+0000 7f6d09819f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-16T23:47:23.819 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:23.816+0000 7f6d09819f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-16T23:47:23.933 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:23.931+0000 7f6d09819f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-16T23:47:24.954 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:47:24.952+0000 7f6ca9aab640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-16T23:47:28.597 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-16T23:47:28.707 INFO:teuthology.misc.health.smithi053.stderr:2023-10-16T23:47:28.701+0000 7f17b551e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:28.710 INFO:teuthology.misc.health.smithi053.stderr:2023-10-16T23:47:28.707+0000 7f17b551e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:28.894 INFO:teuthology.misc.health.smithi053.stdout: 2023-10-16T23:47:28.895 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":9,"fsid":"ec8ab2be-8061-4c8a-9ea6-1163eadae66a","created":"2023-10-16T23:47:19.419081+0000","modified":"2023-10-16T23:47:28.009835+0000","last_up_change":"2023-10-16T23:47:26.003277+0000","last_in_change":"2023-10-16T23:47:21.329690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T23:47:28.007452+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd6c98cf-26a2-4a1a-8d24-e07b286f63c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":2822464177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2822464177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2822464177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":2822464177}]},"public_addr":"172.21.15.53:6803/2822464177","cluster_addr":"172.21.15.53:6802/2822464177","heartbeat_back_addr":"172.21.15.53:6808/2822464177","heartbeat_front_addr":"172.21.15.53:6809/2822464177","state":["exists","up"]},{"osd":1,"uuid":"6e2af387-5606-4858-bcdc-e57480518213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2533707216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":2533707216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2533707216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":2533707216}]},"public_addr":"172.21.15.53:6804/2533707216","cluster_addr":"172.21.15.53:6805/2533707216","heartbeat_back_addr":"172.21.15.53:6810/2533707216","heartbeat_front_addr":"172.21.15.53:6811/2533707216","state":["exists","up"]},{"osd":2,"uuid":"34c131e5-7d98-4a4a-a569-6367d63d959f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":3262369784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":3262369784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3262369784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3262369784}]},"public_addr":"172.21.15.53:6806/3262369784","cluster_addr":"172.21.15.53:6807/3262369784","heartbeat_back_addr":"172.21.15.53:6812/3262369784","heartbeat_front_addr":"172.21.15.53:6813/3262369784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T23:47:28.905 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-16T23:47:28.905 INFO:tasks.ceph:Creating RBD pool 2023-10-16T23:47:28.906 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-16T23:47:29.017 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:29.008+0000 7f54ae06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:29.021 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:29.019+0000 7f54ae06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:30.020 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'rbd' created 2023-10-16T23:47:30.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-16T23:47:30.154 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:30.144+0000 7fef5506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:30.157 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:30.155+0000 7fef5506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.025 INFO:teuthology.orchestra.run.smithi053.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-16T23:47:31.046 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-16T23:47:31.047 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-16T23:47:31.047 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-16T23:47:31.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-16T23:47:31.048 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-16T23:47:31.198 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.192+0000 7fe73ab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.200 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.198+0000 7fe73ab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.433 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:31.443 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2460526312},{"type":"v1","addr":"172.21.15.53:6801","nonce":2460526312}]},"active_addr":"172.21.15.53:6801/2460526312","active_change":"2023-10-16T23:47:23.946700+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":621659726}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":639532828}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":3659546837}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.53:0","nonce":282493844}]}]} 2023-10-16T23:47:31.446 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-16T23:47:31.446 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-16T23:47:31.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T23:47:31.557 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.552+0000 7fbd7b715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.560 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.558+0000 7fbd7b715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.746 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:31.747 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":12,"fsid":"ec8ab2be-8061-4c8a-9ea6-1163eadae66a","created":"2023-10-16T23:47:19.419081+0000","modified":"2023-10-16T23:47:31.020704+0000","last_up_change":"2023-10-16T23:47:26.003277+0000","last_in_change":"2023-10-16T23:47:21.329690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T23:47:28.007452+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-16T23:47:29.243815+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd6c98cf-26a2-4a1a-8d24-e07b286f63c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":2822464177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2822464177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2822464177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":2822464177}]},"public_addr":"172.21.15.53:6803/2822464177","cluster_addr":"172.21.15.53:6802/2822464177","heartbeat_back_addr":"172.21.15.53:6808/2822464177","heartbeat_front_addr":"172.21.15.53:6809/2822464177","state":["exists","up"]},{"osd":1,"uuid":"6e2af387-5606-4858-bcdc-e57480518213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2533707216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":2533707216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2533707216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":2533707216}]},"public_addr":"172.21.15.53:6804/2533707216","cluster_addr":"172.21.15.53:6805/2533707216","heartbeat_back_addr":"172.21.15.53:6810/2533707216","heartbeat_front_addr":"172.21.15.53:6811/2533707216","state":["exists","up"]},{"osd":2,"uuid":"34c131e5-7d98-4a4a-a569-6367d63d959f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":3262369784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":3262369784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3262369784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3262369784}]},"public_addr":"172.21.15.53:6806/3262369784","cluster_addr":"172.21.15.53:6807/3262369784","heartbeat_back_addr":"172.21.15.53:6812/3262369784","heartbeat_front_addr":"172.21.15.53:6813/3262369784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T23:47:31.758 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-16T23:47:31.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-16T23:47:31.869 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.865+0000 7f012c59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:31.872 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:31.870+0000 7f012c59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:32.061 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":13,"fsid":"ec8ab2be-8061-4c8a-9ea6-1163eadae66a","created":"2023-10-16T23:47:19.419081+0000","modified":"2023-10-16T23:47:32.024517+0000","last_up_change":"2023-10-16T23:47:26.003277+0000","last_in_change":"2023-10-16T23:47:21.329690+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-16T23:47:28.007452+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-16T23:47:29.243815+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd6c98cf-26a2-4a1a-8d24-e07b286f63c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6803","nonce":2822464177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2822464177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2822464177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":2822464177}]},"public_addr":"172.21.15.53:6803/2822464177","cluster_addr":"172.21.15.53:6802/2822464177","heartbeat_back_addr":"172.21.15.53:6808/2822464177","heartbeat_front_addr":"172.21.15.53:6809/2822464177","state":["exists","up"]},{"osd":1,"uuid":"6e2af387-5606-4858-bcdc-e57480518213","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2533707216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6805","nonce":2533707216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2533707216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6811","nonce":2533707216}]},"public_addr":"172.21.15.53:6804/2533707216","cluster_addr":"172.21.15.53:6805/2533707216","heartbeat_back_addr":"172.21.15.53:6810/2533707216","heartbeat_front_addr":"172.21.15.53:6811/2533707216","state":["exists","up"]},{"osd":2,"uuid":"34c131e5-7d98-4a4a-a569-6367d63d959f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":3262369784}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6807","nonce":3262369784}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":3262369784}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3262369784}]},"public_addr":"172.21.15.53:6806/3262369784","cluster_addr":"172.21.15.53:6807/3262369784","heartbeat_back_addr":"172.21.15.53:6812/3262369784","heartbeat_front_addr":"172.21.15.53:6813/3262369784","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-16T23:47:32.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-16T23:47:32.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-16T23:47:32.074 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-16T23:47:32.190 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.185+0000 7f4333915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.191 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.185+0000 7fd52d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.191 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.185+0000 7f24dad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.194 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.192+0000 7fd52d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.194 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.192+0000 7f4333915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.194 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.191+0000 7f24dad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.220 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.220 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-16T23:47:32.222 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-16T23:47:32.231 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.231 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-16T23:47:32.343 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.337+0000 7f9b78e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.346 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.344+0000 7f9b78e24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.381+0000 7f1495a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.390 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.388+0000 7f1495a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.406 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.397+0000 7f820322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.409 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.407+0000 7f820322d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.554 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2023-10-16T23:47:32.567 DEBUG:teuthology.parallel:result is None 2023-10-16T23:47:32.582 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.600 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2023-10-16T23:47:32.600 DEBUG:teuthology.parallel:result is None 2023-10-16T23:47:32.605 INFO:teuthology.orchestra.run.smithi053.stdout:34359738374 2023-10-16T23:47:32.618 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2023-10-16T23:47:32.618 DEBUG:teuthology.parallel:result is None 2023-10-16T23:47:32.619 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-16T23:47:32.619 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T23:47:32.769 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.764+0000 7f794d508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.771 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:32.770+0000 7f794d508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:32.954 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:32.954 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-10-16T23:47:32.965 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-16T23:47:32.221008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.037510+0000","last_change":"2023-10-16T23:47:31.174123+0000","last_active":"2023-10-16T23:47:32.037510+0000","last_peered":"2023-10-16T23:47:32.037510+0000","last_clean":"2023-10-16T23:47:32.037510+0000","last_became_active":"2023-10-16T23:47:31.171281+0000","last_became_peered":"2023-10-16T23:47:31.171281+0000","last_unstale":"2023-10-16T23:47:32.037510+0000","last_undegraded":"2023-10-16T23:47:32.037510+0000","last_fullsized":"2023-10-16T23:47:32.037510+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.035219+0000","last_change":"2023-10-16T23:47:31.176904+0000","last_active":"2023-10-16T23:47:32.035219+0000","last_peered":"2023-10-16T23:47:32.035219+0000","last_clean":"2023-10-16T23:47:32.035219+0000","last_became_active":"2023-10-16T23:47:31.163694+0000","last_became_peered":"2023-10-16T23:47:31.163694+0000","last_unstale":"2023-10-16T23:47:32.035219+0000","last_undegraded":"2023-10-16T23:47:32.035219+0000","last_fullsized":"2023-10-16T23:47:32.035219+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.035441+0000","last_change":"2023-10-16T23:47:31.129337+0000","last_active":"2023-10-16T23:47:32.035441+0000","last_peered":"2023-10-16T23:47:32.035441+0000","last_clean":"2023-10-16T23:47:32.035441+0000","last_became_active":"2023-10-16T23:47:31.119893+0000","last_became_peered":"2023-10-16T23:47:31.119893+0000","last_unstale":"2023-10-16T23:47:32.035441+0000","last_undegraded":"2023-10-16T23:47:32.035441+0000","last_fullsized":"2023-10-16T23:47:32.035441+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.033910+0000","last_change":"2023-10-16T23:47:31.151352+0000","last_active":"2023-10-16T23:47:32.033910+0000","last_peered":"2023-10-16T23:47:32.033910+0000","last_clean":"2023-10-16T23:47:32.033910+0000","last_became_active":"2023-10-16T23:47:31.148288+0000","last_became_peered":"2023-10-16T23:47:31.148288+0000","last_unstale":"2023-10-16T23:47:32.033910+0000","last_undegraded":"2023-10-16T23:47:32.033910+0000","last_fullsized":"2023-10-16T23:47:32.033910+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.141342+0000","last_change":"2023-10-16T23:47:31.142038+0000","last_active":"2023-10-16T23:47:31.141342+0000","last_peered":"2023-10-16T23:47:31.141342+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.130392+0000","last_became_peered":"2023-10-16T23:47:31.130392+0000","last_unstale":"2023-10-16T23:47:31.141342+0000","last_undegraded":"2023-10-16T23:47:31.141342+0000","last_fullsized":"2023-10-16T23:47:31.141342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.151246+0000","last_change":"2023-10-16T23:47:31.151896+0000","last_active":"2023-10-16T23:47:31.151246+0000","last_peered":"2023-10-16T23:47:31.151246+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.147011+0000","last_became_peered":"2023-10-16T23:47:31.147011+0000","last_unstale":"2023-10-16T23:47:31.151246+0000","last_undegraded":"2023-10-16T23:47:31.151246+0000","last_fullsized":"2023-10-16T23:47:31.151246+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.112627+0000","last_change":"2023-10-16T23:47:31.113115+0000","last_active":"2023-10-16T23:47:31.112627+0000","last_peered":"2023-10-16T23:47:31.112627+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.109937+0000","last_became_peered":"2023-10-16T23:47:31.109937+0000","last_unstale":"2023-10-16T23:47:31.112627+0000","last_undegraded":"2023-10-16T23:47:31.112627+0000","last_fullsized":"2023-10-16T23:47:31.112627+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.036524+0000","last_change":"2023-10-16T23:47:31.135413+0000","last_active":"2023-10-16T23:47:32.036524+0000","last_peered":"2023-10-16T23:47:32.036524+0000","last_clean":"2023-10-16T23:47:32.036524+0000","last_became_active":"2023-10-16T23:47:31.119949+0000","last_became_peered":"2023-10-16T23:47:31.119949+0000","last_unstale":"2023-10-16T23:47:32.036524+0000","last_undegraded":"2023-10-16T23:47:32.036524+0000","last_fullsized":"2023-10-16T23:47:32.036524+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.033262+0000","last_change":"2023-10-16T23:47:29.043959+0000","last_active":"2023-10-16T23:47:32.033262+0000","last_peered":"2023-10-16T23:47:32.033262+0000","last_clean":"2023-10-16T23:47:32.033262+0000","last_became_active":"2023-10-16T23:47:29.040748+0000","last_became_peered":"2023-10-16T23:47:29.040748+0000","last_unstale":"2023-10-16T23:47:32.033262+0000","last_undegraded":"2023-10-16T23:47:32.033262+0000","last_fullsized":"2023-10-16T23:47:32.033262+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:28.009835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:28.009835+0000","last_clean_scrub_stamp":"2023-10-16T23:47:28.009835+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-16T23:47:32.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-16T23:47:33.077 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:33.073+0000 7f32c6915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:33.080 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:33.079+0000 7f32c6915640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:33.268 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:33.268 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2023-10-16T23:47:33.279 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-10-16T23:47:32.221008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82548,"kb_used_data":1316,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490252,"statfs":{"total":322122547200,"available":322038018048,"internally_reserved":0,"allocated":1347584,"data_stored":1230629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.037510+0000","last_change":"2023-10-16T23:47:31.174123+0000","last_active":"2023-10-16T23:47:32.037510+0000","last_peered":"2023-10-16T23:47:32.037510+0000","last_clean":"2023-10-16T23:47:32.037510+0000","last_became_active":"2023-10-16T23:47:31.171281+0000","last_became_peered":"2023-10-16T23:47:31.171281+0000","last_unstale":"2023-10-16T23:47:32.037510+0000","last_undegraded":"2023-10-16T23:47:32.037510+0000","last_fullsized":"2023-10-16T23:47:32.037510+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.035219+0000","last_change":"2023-10-16T23:47:31.176904+0000","last_active":"2023-10-16T23:47:32.035219+0000","last_peered":"2023-10-16T23:47:32.035219+0000","last_clean":"2023-10-16T23:47:32.035219+0000","last_became_active":"2023-10-16T23:47:31.163694+0000","last_became_peered":"2023-10-16T23:47:31.163694+0000","last_unstale":"2023-10-16T23:47:32.035219+0000","last_undegraded":"2023-10-16T23:47:32.035219+0000","last_fullsized":"2023-10-16T23:47:32.035219+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.035441+0000","last_change":"2023-10-16T23:47:31.129337+0000","last_active":"2023-10-16T23:47:32.035441+0000","last_peered":"2023-10-16T23:47:32.035441+0000","last_clean":"2023-10-16T23:47:32.035441+0000","last_became_active":"2023-10-16T23:47:31.119893+0000","last_became_peered":"2023-10-16T23:47:31.119893+0000","last_unstale":"2023-10-16T23:47:32.035441+0000","last_undegraded":"2023-10-16T23:47:32.035441+0000","last_fullsized":"2023-10-16T23:47:32.035441+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.033910+0000","last_change":"2023-10-16T23:47:31.151352+0000","last_active":"2023-10-16T23:47:32.033910+0000","last_peered":"2023-10-16T23:47:32.033910+0000","last_clean":"2023-10-16T23:47:32.033910+0000","last_became_active":"2023-10-16T23:47:31.148288+0000","last_became_peered":"2023-10-16T23:47:31.148288+0000","last_unstale":"2023-10-16T23:47:32.033910+0000","last_undegraded":"2023-10-16T23:47:32.033910+0000","last_fullsized":"2023-10-16T23:47:32.033910+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.141342+0000","last_change":"2023-10-16T23:47:31.142038+0000","last_active":"2023-10-16T23:47:31.141342+0000","last_peered":"2023-10-16T23:47:31.141342+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.130392+0000","last_became_peered":"2023-10-16T23:47:31.130392+0000","last_unstale":"2023-10-16T23:47:31.141342+0000","last_undegraded":"2023-10-16T23:47:31.141342+0000","last_fullsized":"2023-10-16T23:47:31.141342+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.151246+0000","last_change":"2023-10-16T23:47:31.151896+0000","last_active":"2023-10-16T23:47:31.151246+0000","last_peered":"2023-10-16T23:47:31.151246+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.147011+0000","last_became_peered":"2023-10-16T23:47:31.147011+0000","last_unstale":"2023-10-16T23:47:31.151246+0000","last_undegraded":"2023-10-16T23:47:31.151246+0000","last_fullsized":"2023-10-16T23:47:31.151246+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-16T23:47:31.112627+0000","last_change":"2023-10-16T23:47:31.113115+0000","last_active":"2023-10-16T23:47:31.112627+0000","last_peered":"2023-10-16T23:47:31.112627+0000","last_clean":"2023-10-16T23:47:30.017078+0000","last_became_active":"2023-10-16T23:47:31.109937+0000","last_became_peered":"2023-10-16T23:47:31.109937+0000","last_unstale":"2023-10-16T23:47:31.112627+0000","last_undegraded":"2023-10-16T23:47:31.112627+0000","last_fullsized":"2023-10-16T23:47:31.112627+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.036524+0000","last_change":"2023-10-16T23:47:31.135413+0000","last_active":"2023-10-16T23:47:32.036524+0000","last_peered":"2023-10-16T23:47:32.036524+0000","last_clean":"2023-10-16T23:47:32.036524+0000","last_became_active":"2023-10-16T23:47:31.119949+0000","last_became_peered":"2023-10-16T23:47:31.119949+0000","last_unstale":"2023-10-16T23:47:32.036524+0000","last_undegraded":"2023-10-16T23:47:32.036524+0000","last_fullsized":"2023-10-16T23:47:32.036524+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:30.017078+0000","last_clean_scrub_stamp":"2023-10-16T23:47:30.017078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-16T23:47:32.033262+0000","last_change":"2023-10-16T23:47:29.043959+0000","last_active":"2023-10-16T23:47:32.033262+0000","last_peered":"2023-10-16T23:47:32.033262+0000","last_clean":"2023-10-16T23:47:32.033262+0000","last_became_active":"2023-10-16T23:47:29.040748+0000","last_became_peered":"2023-10-16T23:47:29.040748+0000","last_unstale":"2023-10-16T23:47:32.033262+0000","last_undegraded":"2023-10-16T23:47:32.033262+0000","last_fullsized":"2023-10-16T23:47:32.033262+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-16T23:47:28.009835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-16T23:47:28.009835+0000","last_clean_scrub_stamp":"2023-10-16T23:47:28.009835+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27004,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830596,"statfs":{"total":107374182400,"available":107346530304,"internally_reserved":0,"allocated":53248,"data_stored":16631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27772,"kb_used_data":632,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829828,"statfs":{"total":107374182400,"available":107345743872,"internally_reserved":0,"allocated":647168,"data_stored":606999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-16T23:47:33.280 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-16T23:47:33.281 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-16T23:47:33.281 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-16T23:47:33.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-16T23:47:33.391 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:33.387+0000 7f6807c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:33.394 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-16T23:47:33.393+0000 7f6807c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-16T23:47:33.614 INFO:teuthology.orchestra.run.smithi053.stdout: 2023-10-16T23:47:33.615 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-16T23:47:33.626 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-16T23:47:33.627 INFO:teuthology.run_tasks:Running task workunit... 2023-10-16T23:47:33.668 INFO:tasks.workunit:Pulling workunits from ref 06f14dfbeeb0dce341775222d82ea9644e37c496 2023-10-16T23:47:33.669 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-16T23:47:33.669 DEBUG:teuthology.orchestra.run.smithi053:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-16T23:47:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-16T23:47:33.685 INFO:teuthology.orchestra.run.smithi053.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-16T23:47:33.685 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-16T23:47:33.739 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-16T23:47:33.740 DEBUG:teuthology.orchestra.run.smithi053:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-16T23:47:33.795 INFO:tasks.workunit:timeout=3h 2023-10-16T23:47:33.795 INFO:tasks.workunit:cleanup=True 2023-10-16T23:47:33.795 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 06f14dfbeeb0dce341775222d82ea9644e37c496 2023-10-16T23:47:33.850 INFO:tasks.workunit.client.0.smithi053.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-16T23:48:20.665 INFO:tasks.workunit.client.0.smithi053.stderr:Note: switching to '06f14dfbeeb0dce341775222d82ea9644e37c496'. 2023-10-16T23:48:20.665 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.666 INFO:tasks.workunit.client.0.smithi053.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-16T23:48:20.666 INFO:tasks.workunit.client.0.smithi053.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-16T23:48:20.666 INFO:tasks.workunit.client.0.smithi053.stderr:state without impacting any branches by switching back to a branch. 2023-10-16T23:48:20.666 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.666 INFO:tasks.workunit.client.0.smithi053.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr: git switch -c 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr:Or undo this operation with: 2023-10-16T23:48:20.667 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.668 INFO:tasks.workunit.client.0.smithi053.stderr: git switch - 2023-10-16T23:48:20.668 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.668 INFO:tasks.workunit.client.0.smithi053.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-16T23:48:20.668 INFO:tasks.workunit.client.0.smithi053.stderr: 2023-10-16T23:48:20.668 INFO:tasks.workunit.client.0.smithi053.stderr:HEAD is now at 06f14dfbeeb Merge branches sjust/wip-crimson-scrub 2023-10-16T23:48:20.669 DEBUG:teuthology.orchestra.run.smithi053:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-16T23:48:20.738 INFO:tasks.workunit.client.0.smithi053.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-16T23:48:20.741 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-16T23:48:20.741 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-16T23:48:21.430 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-16T23:48:21.506 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-16T23:48:21.549 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-16T23:48:21.550 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-16T23:48:21.550 INFO:tasks.workunit.client.0.smithi053.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-16T23:48:21.592 INFO:tasks.workunit.client.0.smithi053.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-16T23:48:21.595 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2023-10-16T23:48:21.595 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-16T23:48:21.651 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2023-10-16T23:48:21.652 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2023-10-16T23:48:21.653 DEBUG:teuthology.orchestra.run.smithi053:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=06f14dfbeeb0dce341775222d82ea9644e37c496 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2023-10-16T23:48:21.726 INFO:tasks.workunit.client.0.smithi053.stdout:Running main() from gmock_main.cc 2023-10-16T23:48:21.726 INFO:tasks.workunit.client.0.smithi053.stdout:[==========] Running 70 tests from 1 test suite. 2023-10-16T23:48:21.727 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] Global test environment set-up. 2023-10-16T23:48:21.727 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] 70 tests from TestClsRbd 2023-10-16T23:48:23.437 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_all_features 2023-10-16T23:48:23.928 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_all_features (491 ms) 2023-10-16T23:48:23.929 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.copyup 2023-10-16T23:48:24.163 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.copyup (235 ms) 2023-10-16T23:48:24.164 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.sparse_copyup 2023-10-16T23:48:24.243 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.sparse_copyup (79 ms) 2023-10-16T23:48:24.243 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_and_set_id 2023-10-16T23:48:24.298 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_and_set_id (56 ms) 2023-10-16T23:48:24.299 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.add_remove_child 2023-10-16T23:48:24.384 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.add_remove_child (86 ms) 2023-10-16T23:48:24.385 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.directory_methods 2023-10-16T23:48:24.637 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.directory_methods (253 ms) 2023-10-16T23:48:24.638 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.create 2023-10-16T23:48:24.739 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.create (102 ms) 2023-10-16T23:48:24.739 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_features 2023-10-16T23:48:24.751 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_features (11 ms) 2023-10-16T23:48:24.751 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_object_prefix 2023-10-16T23:48:24.762 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_object_prefix (12 ms) 2023-10-16T23:48:24.762 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2023-10-16T23:48:24.772 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_create_timestamp (10 ms) 2023-10-16T23:48:24.773 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2023-10-16T23:48:24.782 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_access_timestamp (10 ms) 2023-10-16T23:48:24.783 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2023-10-16T23:48:24.793 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_modify_timestamp (11 ms) 2023-10-16T23:48:24.793 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_data_pool 2023-10-16T23:48:24.818 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_data_pool (25 ms) 2023-10-16T23:48:24.818 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.get_size 2023-10-16T23:48:24.849 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.get_size (30 ms) 2023-10-16T23:48:24.849 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.set_size 2023-10-16T23:48:24.889 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.set_size (38 ms) 2023-10-16T23:48:24.889 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.protection_status 2023-10-16T23:48:25.082 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.protection_status (196 ms) 2023-10-16T23:48:25.082 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.snapshot_limits 2023-10-16T23:48:25.142 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.snapshot_limits (60 ms) 2023-10-16T23:48:25.142 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.parents_v1 2023-10-16T23:48:25.497 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.parents_v1 (355 ms) 2023-10-16T23:48:25.498 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.parents_v2 2023-10-16T23:48:25.739 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.parents_v2 (242 ms) 2023-10-16T23:48:25.740 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.snapshots 2023-10-16T23:48:25.928 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.snapshots (189 ms) 2023-10-16T23:48:25.929 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.snapid_race 2023-10-16T23:48:25.958 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.snapid_race (30 ms) 2023-10-16T23:48:25.959 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.stripingv2 2023-10-16T23:48:26.033 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.stripingv2 (74 ms) 2023-10-16T23:48:26.033 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_save 2023-10-16T23:48:26.043 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_save (11 ms) 2023-10-16T23:48:26.043 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_resize 2023-10-16T23:48:26.086 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_resize (43 ms) 2023-10-16T23:48:26.086 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_update 2023-10-16T23:48:26.117 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_update (31 ms) 2023-10-16T23:48:26.117 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2023-10-16T23:48:26.119 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_load_enoent (1 ms) 2023-10-16T23:48:26.119 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2023-10-16T23:48:26.146 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_snap_add (28 ms) 2023-10-16T23:48:26.146 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2023-10-16T23:48:26.174 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.object_map_snap_remove (28 ms) 2023-10-16T23:48:26.175 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.flags 2023-10-16T23:48:26.214 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.flags (40 ms) 2023-10-16T23:48:26.215 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.metadata 2023-10-16T23:48:26.413 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.metadata (199 ms) 2023-10-16T23:48:26.414 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.set_features 2023-10-16T23:48:26.468 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.set_features (54 ms) 2023-10-16T23:48:26.468 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror 2023-10-16T23:48:26.744 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror (276 ms) 2023-10-16T23:48:26.744 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror_image 2023-10-16T23:48:26.869 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror_image (125 ms) 2023-10-16T23:48:26.869 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror_image_status 2023-10-16T23:49:00.573 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror_image_status (33703 ms) 2023-10-16T23:49:00.573 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror_image_map 2023-10-16T23:49:01.887 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror_image_map (1314 ms) 2023-10-16T23:49:01.887 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror_instances 2023-10-16T23:49:01.947 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror_instances (60 ms) 2023-10-16T23:49:01.947 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2023-10-16T23:49:02.043 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.mirror_snapshot (95 ms) 2023-10-16T23:49:02.043 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_dir_list 2023-10-16T23:49:02.065 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_dir_list (22 ms) 2023-10-16T23:49:02.065 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_dir_add 2023-10-16T23:49:02.082 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_dir_add (17 ms) 2023-10-16T23:49:02.082 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2023-10-16T23:49:02.105 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.dir_add_already_existing (23 ms) 2023-10-16T23:49:02.105 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_dir_rename 2023-10-16T23:49:02.153 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_dir_rename (48 ms) 2023-10-16T23:49:02.154 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_dir_remove 2023-10-16T23:49:02.178 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_dir_remove (25 ms) 2023-10-16T23:49:02.178 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2023-10-16T23:49:02.210 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (32 ms) 2023-10-16T23:49:02.210 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_image_add 2023-10-16T23:49:02.225 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_image_add (15 ms) 2023-10-16T23:49:02.226 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_image_remove 2023-10-16T23:49:02.252 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_image_remove (26 ms) 2023-10-16T23:49:02.252 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_image_list 2023-10-16T23:49:02.272 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_image_list (21 ms) 2023-10-16T23:49:02.272 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_image_clean 2023-10-16T23:49:02.304 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_image_clean (31 ms) 2023-10-16T23:49:02.304 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.image_group_add 2023-10-16T23:49:02.322 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.image_group_add (18 ms) 2023-10-16T23:49:02.322 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.image_group_remove 2023-10-16T23:49:02.347 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.image_group_remove (26 ms) 2023-10-16T23:49:02.347 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.image_group_get 2023-10-16T23:49:02.365 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.image_group_get (18 ms) 2023-10-16T23:49:02.366 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2023-10-16T23:49:02.379 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (14 ms) 2023-10-16T23:49:02.379 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2023-10-16T23:49:02.393 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (14 ms) 2023-10-16T23:49:02.393 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2023-10-16T23:49:02.414 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (21 ms) 2023-10-16T23:49:02.415 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2023-10-16T23:49:02.448 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (34 ms) 2023-10-16T23:49:02.448 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_set 2023-10-16T23:49:02.465 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_set (16 ms) 2023-10-16T23:49:02.465 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_list 2023-10-16T23:49:02.489 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_list (25 ms) 2023-10-16T23:49:02.489 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2023-10-16T23:49:02.617 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (127 ms) 2023-10-16T23:49:02.617 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2023-10-16T23:49:04.119 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (1503 ms) 2023-10-16T23:49:04.119 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_remove 2023-10-16T23:49:04.143 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_remove (24 ms) 2023-10-16T23:49:04.144 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2023-10-16T23:49:04.159 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (16 ms) 2023-10-16T23:49:04.159 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.trash_methods 2023-10-16T23:49:04.220 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.trash_methods (61 ms) 2023-10-16T23:49:04.220 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.op_features 2023-10-16T23:49:04.273 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.op_features (53 ms) 2023-10-16T23:49:04.274 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.clone_parent 2023-10-16T23:49:04.443 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.clone_parent (169 ms) 2023-10-16T23:49:04.443 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2023-10-16T23:49:04.518 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.clone_parent_ns (75 ms) 2023-10-16T23:49:04.518 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.clone_child 2023-10-16T23:49:04.609 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.clone_child (92 ms) 2023-10-16T23:49:04.609 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.namespace_methods 2023-10-16T23:49:04.665 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.namespace_methods (56 ms) 2023-10-16T23:49:04.666 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.migration 2023-10-16T23:49:04.750 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.migration (84 ms) 2023-10-16T23:49:04.750 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.migration_v1 2023-10-16T23:49:04.816 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.migration_v1 (66 ms) 2023-10-16T23:49:04.816 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2023-10-16T23:49:06.472 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1657 ms) 2023-10-16T23:49:06.472 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] TestClsRbd.sparsify 2023-10-16T23:49:06.677 INFO:tasks.workunit.client.0.smithi053.stdout:[ OK ] TestClsRbd.sparsify (204 ms) 2023-10-16T23:49:07.474 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] 70 tests from TestClsRbd (43239 ms total) 2023-10-16T23:49:07.474 INFO:tasks.workunit.client.0.smithi053.stdout: 2023-10-16T23:49:07.475 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] Global test environment tear-down 2023-10-16T23:49:07.475 INFO:tasks.workunit.client.0.smithi053.stdout:[==========] 70 tests from 1 test suite ran. (45747 ms total) 2023-10-16T23:49:07.475 INFO:tasks.workunit.client.0.smithi053.stdout:[ PASSED ] 70 tests. 2023-10-16T23:49:07.477 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-16T23:49:07.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-16T23:49:07.549 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2023-10-16T23:49:07.551 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2023-10-16T23:49:07.551 DEBUG:teuthology.orchestra.run.smithi053:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=06f14dfbeeb0dce341775222d82ea9644e37c496 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2023-10-16T23:49:07.625 INFO:tasks.workunit.client.0.smithi053.stdout:Running main() from gmock_main.cc 2023-10-16T23:49:07.625 INFO:tasks.workunit.client.0.smithi053.stdout:[==========] Running 11 tests from 1 test suite. 2023-10-16T23:49:07.626 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] Global test environment set-up. 2023-10-16T23:49:07.626 INFO:tasks.workunit.client.0.smithi053.stdout:[----------] 11 tests from ClsLock 2023-10-16T23:49:07.626 INFO:tasks.workunit.client.0.smithi053.stdout:[ RUN ] ClsLock.TestMultiLocking 2023-10-16T23:49:10.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-16T23:49:15.814 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-16T23:49:15.815 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-10-16T23:49:21.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2023-10-16T23:49:21.118 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2023-10-16T23:49:26.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2023-10-16T23:49:26.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2023-10-16T23:49:31.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2023-10-16T23:49:31.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~21s 2023-10-16T23:49:37.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2023-10-16T23:49:37.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2023-10-16T23:49:42.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2023-10-16T23:49:42.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2023-10-16T23:49:47.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2023-10-16T23:49:47.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~37s 2023-10-16T23:49:52.938 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2023-10-16T23:49:52.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~43s 2023-10-16T23:49:58.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2023-10-16T23:49:58.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~48s 2023-10-16T23:50:03.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2023-10-16T23:50:03.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~53s 2023-10-16T23:50:08.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2023-10-16T23:50:08.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~58s 2023-10-16T23:50:09.423 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:09.422+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:14.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2023-10-16T23:50:14.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~64s 2023-10-16T23:50:14.425 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:14.423+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:19.425 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:19.424+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:19.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2023-10-16T23:50:19.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~69s 2023-10-16T23:50:24.427 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:24.425+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:24.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2023-10-16T23:50:24.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~74s 2023-10-16T23:50:29.428 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:29.426+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:30.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2023-10-16T23:50:30.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2023-10-16T23:50:34.429 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:34.427+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:35.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2023-10-16T23:50:35.361 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2023-10-16T23:50:39.430 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:39.428+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:40.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2023-10-16T23:50:40.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2023-10-16T23:50:44.432 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:44.430+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:45.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2023-10-16T23:50:45.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~96s 2023-10-16T23:50:49.433 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:49.430+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2023-10-16T23:50:51.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~101s 2023-10-16T23:50:54.434 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:54.431+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:50:56.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2023-10-16T23:50:56.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~106s 2023-10-16T23:50:59.435 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:50:59.433+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:01.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~106s 2023-10-16T23:51:01.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~111s 2023-10-16T23:51:04.436 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:04.435+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:07.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~111s 2023-10-16T23:51:07.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~117s 2023-10-16T23:51:09.438 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:09.436+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:12.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2023-10-16T23:51:12.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~122s 2023-10-16T23:51:14.438 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:14.437+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:17.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2023-10-16T23:51:17.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2023-10-16T23:51:19.439 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:19.438+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:23.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2023-10-16T23:51:23.086 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~133s 2023-10-16T23:51:24.441 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:24.440+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:28.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2023-10-16T23:51:28.389 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2023-10-16T23:51:29.442 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:29.440+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:33.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2023-10-16T23:51:33.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2023-10-16T23:51:34.443 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:34.441+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:38.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2023-10-16T23:51:38.994 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2023-10-16T23:51:39.444 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:39.442+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:44.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~148s 2023-10-16T23:51:44.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2023-10-16T23:51:44.445 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:44.443+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:49.446 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:49.444+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:49.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2023-10-16T23:51:49.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~159s 2023-10-16T23:51:54.447 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:54.446+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:51:54.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~159s 2023-10-16T23:51:54.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~164s 2023-10-16T23:51:59.448 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:51:59.447+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:00.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~164s 2023-10-16T23:52:00.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2023-10-16T23:52:04.450 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:04.448+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:05.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2023-10-16T23:52:05.512 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2023-10-16T23:52:09.451 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:09.449+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:10.815 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2023-10-16T23:52:10.815 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2023-10-16T23:52:14.452 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:14.450+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:16.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2023-10-16T23:52:16.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~186s 2023-10-16T23:52:19.453 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:19.451+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:21.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2023-10-16T23:52:21.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~191s 2023-10-16T23:52:24.454 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:24.453+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:26.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2023-10-16T23:52:26.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~196s 2023-10-16T23:52:29.455 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:29.453+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:32.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~196s 2023-10-16T23:52:32.030 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~202s 2023-10-16T23:52:34.456 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:34.453+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:37.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2023-10-16T23:52:37.334 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~207s 2023-10-16T23:52:39.457 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:39.454+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:42.636 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2023-10-16T23:52:42.637 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~212s 2023-10-16T23:52:44.458 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:44.457+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:47.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~212s 2023-10-16T23:52:47.939 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~218s 2023-10-16T23:52:49.460 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:49.458+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:53.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~217s 2023-10-16T23:52:53.242 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~223s 2023-10-16T23:52:54.461 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:54.459+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:52:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2023-10-16T23:52:58.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~228s 2023-10-16T23:52:59.462 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:52:59.460+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:03.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~228s 2023-10-16T23:53:03.847 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~233s 2023-10-16T23:53:04.463 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:04.462+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:09.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~233s 2023-10-16T23:53:09.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~239s 2023-10-16T23:53:09.464 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:09.463+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2023-10-16T23:53:14.453 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~244s 2023-10-16T23:53:14.465 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:14.464+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:19.466 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:19.465+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:19.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2023-10-16T23:53:19.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~249s 2023-10-16T23:53:24.468 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:24.466+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:25.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~249s 2023-10-16T23:53:25.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2023-10-16T23:53:29.469 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:29.466+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:30.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2023-10-16T23:53:30.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2023-10-16T23:53:34.470 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:34.468+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:35.664 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2023-10-16T23:53:35.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2023-10-16T23:53:39.471 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:39.469+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:40.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~265s 2023-10-16T23:53:40.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~271s 2023-10-16T23:53:44.472 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:44.471+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:46.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2023-10-16T23:53:46.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2023-10-16T23:53:49.473 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:49.472+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:51.573 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2023-10-16T23:53:51.574 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2023-10-16T23:53:54.474 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:54.472+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:53:56.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2023-10-16T23:53:56.878 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2023-10-16T23:53:59.475 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:53:59.473+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:54:02.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~286s 2023-10-16T23:54:02.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2023-10-16T23:54:04.476 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:54:04.475+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:54:07.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2023-10-16T23:54:07.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2023-10-16T23:54:09.477 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:54:09.476+0000 7f85905f6640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.53:6803/2822464177 for 29sec e21 v21) 2023-10-16T23:54:12.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2023-10-16T23:54:12.787 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~302s 2023-10-16T23:54:12.787 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-10-16T23:54:13.090 INFO:tasks.ceph.osd.1:Sent signal 15 2023-10-16T23:54:13.091 INFO:tasks.ceph.mon.a:Sent signal 15 2023-10-16T23:54:13.091 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-10-16T23:54:13.092 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:54:13.090+0000 7f85935fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 82757) UID: 0 2023-10-16T23:54:13.092 INFO:tasks.ceph.mon.a.smithi053.stderr:2023-10-16T23:54:13.090+0000 7f85935fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-16T23:54:13.093 INFO:tasks.ceph.mgr.x.smithi053.stderr:2023-10-16T23:54:13.091+0000 7f6d03bf7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 82759) UID: 0 2023-10-17T02:49:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-17T02:49:07.633 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh'] on client.0... 2023-10-17T02:49:07.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-17T02:49:07.942 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_lock.sh) on smithi053 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=06f14dfbeeb0dce341775222d82ea9644e37c496 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh' 2023-10-17T02:49:08.013 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a1891fa048a34883837698357c050253 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_lock.sh) on smithi053 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=06f14dfbeeb0dce341775222d82ea9644e37c496 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh' 2023-10-17T02:49:08.016 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-17T02:49:08.026 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-17T02:49:08.026 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-17T02:49:08.142 INFO:teuthology.orchestra.run.smithi053.stderr:2023-10-17T02:49:08.136+0000 7f4b618d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-17T02:51:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-10-17T02:51:08.068 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T02:51:08.077 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-17T02:51:08.077 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-17T02:51:08.078 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-17T02:51:08.079 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:51:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:08.081 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2023-10-17T02:51:08.091 INFO:tasks.ceph.osd.0:Stopped 2023-10-17T02:51:08.091 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-17T02:51:08.092 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:51:08.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:08.093 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2023-10-17T02:51:08.102 INFO:tasks.ceph.osd.1:Stopped 2023-10-17T02:51:08.102 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-17T02:51:08.103 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:51:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:08.104 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2023-10-17T02:51:08.112 INFO:tasks.ceph.osd.2:Stopped 2023-10-17T02:51:08.112 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-17T02:51:08.113 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-17T02:51:08.113 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:51:08.114 INFO:tasks.ceph.mgr.x:Stopped 2023-10-17T02:51:08.115 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-17T02:51:08.115 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-17T02:51:08.115 INFO:teuthology.orchestra.run:waiting for 300 2023-10-17T02:51:08.116 INFO:tasks.ceph.mon.a:Stopped 2023-10-17T02:51:08.117 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-17T02:51:08.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-17T02:51:08.167 INFO:teuthology.orchestra.run.smithi053.stdout:2023-10-16T23:50:10.530930+0000 mon.a (mon.0) 129 : cluster [WRN] Health check failed: Reduced data availability: 32 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2023-10-17T02:51:08.168 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-17T02:51:08.169 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-17T02:51:08.169 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-17T02:51:08.275 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-17T02:51:08.276 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-17T02:51:08.349 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi053.front.sepia.ceph.com 2023-10-17T02:51:08.350 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-17T02:51:08.429 INFO:tasks.ceph:Archiving mon data... 2023-10-17T02:51:08.431 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682/data/mon.a.tgz 2023-10-17T02:51:08.432 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2023-10-17T02:51:08.448 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.ciYQI2vegz 2023-10-17T02:51:08.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ciYQI2vegz 2023-10-17T02:51:08.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.ciYQI2vegz 2023-10-17T02:51:08.711 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.ciYQI2vegz is 214KB 2023-10-17T02:51:08.805 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.ciYQI2vegz 2023-10-17T02:51:08.820 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-17T02:51:08.820 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-17T02:51:08.890 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-17T02:51:08.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-17T02:51:08.963 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-17T02:51:08.968 INFO:tasks.ceph:Archiving crash dumps... 2023-10-17T02:51:08.970 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682/remote/smithi053/crash 2023-10-17T02:51:08.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-17T02:51:09.038 INFO:tasks.ceph:Compressing logs... 2023-10-17T02:51:09.039 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-17T02:51:09.109 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84723.log 2023-10-17T02:51:09.109 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-17T02:51:09.110 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83224.log 2023-10-17T02:51:09.119 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84723.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84723.log.gz 2023-10-17T02:51:09.119 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82999.log 2023-10-17T02:51:09.120 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.83224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84681.log 2023-10-17T02:51:09.120 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83224.log.gz 2023-10-17T02:51:09.120 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.82999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84847.log 2023-10-17T02:51:09.120 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82999.log.gz 2023-10-17T02:51:09.121 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85417.log 2023-10-17T02:51:09.121 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84681.log.gz 2023-10-17T02:51:09.121 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84578.log 2023-10-17T02:51:09.121 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84847.log.gz 2023-10-17T02:51:09.122 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84522.log 2023-10-17T02:51:09.122 INFO:teuthology.orchestra.run.smithi053.stderr: 63.9% -- replaced with /var/log/ceph/ceph-client.admin.85417.log.gz 2023-10-17T02:51:09.122 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83196.log 2023-10-17T02:51:09.123 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84578.log.gz 2023-10-17T02:51:09.123 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84522.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-17T02:51:09.123 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84522.log.gz 2023-10-17T02:51:09.123 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.83196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82849.log 2023-10-17T02:51:09.124 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83196.log.gz 2023-10-17T02:51:09.124 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85059.log 2023-10-17T02:51:09.124 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.82849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84845.log 2023-10-17T02:51:09.125 INFO:teuthology.orchestra.run.smithi053.stderr: 85.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-17T02:51:09.125 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82849.log.gz 2023-10-17T02:51:09.125 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84765.log 2023-10-17T02:51:09.125 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85059.log.gz 2023-10-17T02:51:09.126 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82959.log 2023-10-17T02:51:09.126 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84845.log.gz 2023-10-17T02:51:09.126 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log/var/log/ceph/ceph-client.admin.84765.log: 2023-10-17T02:51:09.126 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84765.log.gz 2023-10-17T02:51:09.127 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.82959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83039.log 2023-10-17T02:51:09.127 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82959.log.gz 2023-10-17T02:51:09.127 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85143.log 2023-10-17T02:51:09.127 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.83039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84846.log 2023-10-17T02:51:09.128 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83039.log.gz 2023-10-17T02:51:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85143.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-17T02:51:09.128 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.85143.log.gz 2023-10-17T02:51:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84972.log 49.6% 2023-10-17T02:51:09.129 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.84846.log.gz 2023-10-17T02:51:09.129 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-mgr.x.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.84639.log 2023-10-17T02:51:09.129 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85784.log 2023-10-17T02:51:09.130 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84972.log.gz 2023-10-17T02:51:09.130 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-17T02:51:09.130 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84639.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84639.log.gz 2023-10-17T02:51:09.130 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84973.log 2023-10-17T02:51:09.131 INFO:teuthology.orchestra.run.smithi053.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.85784.log.gz 2023-10-17T02:51:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-17T02:51:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84973.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-17T02:51:09.131 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84973.log.gz 2023-10-17T02:51:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85361.log 2023-10-17T02:51:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82761.log 2023-10-17T02:51:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85361.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79511.log 2023-10-17T02:51:09.132 INFO:teuthology.orchestra.run.smithi053.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85361.log.gz 2023-10-17T02:51:09.133 INFO:teuthology.orchestra.run.smithi053.stderr: 90.5%/var/log/ceph/ceph-client.admin.82761.log: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-17T02:51:09.133 INFO:teuthology.orchestra.run.smithi053.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82761.log.gz 2023-10-17T02:51:09.133 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.85101.log 2023-10-17T02:51:09.133 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.tmp-client.admin.79511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84933.log 2023-10-17T02:51:09.134 INFO:teuthology.orchestra.run.smithi053.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79511.log.gz 2023-10-17T02:51:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.85101.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.85101.log.gz 2023-10-17T02:51:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.84933.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84933.log.gz 2023-10-17T02:51:09.151 INFO:teuthology.orchestra.run.smithi053.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-17T02:51:09.198 INFO:teuthology.orchestra.run.smithi053.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-17T02:51:09.218 INFO:teuthology.orchestra.run.smithi053.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-17T02:51:09.299 INFO:teuthology.orchestra.run.smithi053.stderr: 95.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-17T02:51:09.399 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-17T02:51:09.401 INFO:teuthology.orchestra.run.smithi053.stderr: 2023-10-17T02:51:09.401 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.307s 2023-10-17T02:51:09.401 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m0.701s 2023-10-17T02:51:09.402 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.052s 2023-10-17T02:51:09.402 INFO:tasks.ceph:Archiving logs... 2023-10-17T02:51:09.403 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682/remote/smithi053/log 2023-10-17T02:51:09.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-17T02:51:09.623 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T02:51:09.633 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-17T02:51:09.646 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T02:51:09.650 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-17T02:51:09.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-17T02:51:09.711 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-17T02:51:09.720 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-17T02:51:09.721 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-17T02:51:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2023-10-17T02:51:09.744 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-17T02:51:09.745 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2023-10-17T02:51:09.745 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 -309us[ -309us] +/- 79ms 2023-10-17T02:51:09.745 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 8 377 20 -2081us[-2054us] +/- 53ms 2023-10-17T02:51:09.746 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 80 +3643us[+3673us] +/- 97ms 2023-10-17T02:51:09.746 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-17T02:51:09.747 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-17T02:51:09.757 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-17T02:51:09.758 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-17T02:51:09.796 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-17T02:51:09.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-17T02:51:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:09.932 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-17T02:51:09.942 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-17T02:51:09.979 INFO:teuthology.task.internal:Duration was 11606.329314 seconds 2023-10-17T02:51:09.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-17T02:51:10.031 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-17T02:51:10.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-17T02:51:10.082 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-17T02:51:10.395 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-17T02:51:10.396 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2023-10-17T02:51:10.396 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-17T02:51:10.416 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-17T02:51:10.417 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-17T02:51:10.488 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-17T02:51:10.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-17T02:51:10.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-17T02:51:10.642 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-17T02:51:10.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-17T02:51:10.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-17T02:51:10.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-17T02:51:10.743 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2023-10-17T02:51:10.769 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-17T02:51:10.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:10.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-17T02:51:10.835 INFO:teuthology.task.internal:Transferring archived files... 2023-10-17T02:51:10.836 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682/remote/smithi053 2023-10-17T02:51:10.836 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-17T02:51:10.904 INFO:teuthology.task.internal:Removing archive directory... 2023-10-17T02:51:10.905 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-17T02:51:10.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-17T02:51:10.966 INFO:teuthology.task.internal:Not uploading archives. 2023-10-17T02:51:10.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-17T02:51:10.997 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-17T02:51:10.998 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-17T02:51:11.012 INFO:teuthology.orchestra.run.smithi053.stdout: 265242 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 17 02:51 /home/ubuntu/cephtest 2023-10-17T02:51:11.013 INFO:teuthology.orchestra.run.smithi053.stdout: 531677 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 16 23:47 /home/ubuntu/cephtest/mnt.0 2023-10-17T02:51:11.013 INFO:teuthology.orchestra.run.smithi053.stdout: 531678 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 16 23:49 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-17T02:51:11.013 INFO:teuthology.orchestra.run.smithi053.stdout: 545588 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 16 23:49 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-17T02:51:11.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-17T02:51:11.015 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-17T02:51:11.016 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_06f14dfbeeb0dce341775222d82ea9644e37c496/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-17T02:51:11.027 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-17T02:51:11.047 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-17T02:51:11.088 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-10-17T02:51:11.088 INFO:teuthology.nuke:Checking targets against current locks 2023-10-17T02:51:11.117 DEBUG:teuthology.nuke:shortname: smithi053 2023-10-17T02:51:11.117 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-17T02:51:11.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sjust-2023-10-16_23:13:53-crimson-rados-wip-crimson-scrub-testing-2023-10-15-distro-default-smithi/7430682', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-16 23:29:45.616075', 'locked_by': 'scheduled_sjust@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeOtfSQGEi39/FCzuey/qkBwXw9MuQMnDkE0Bc7f2K1cUAKQ8bY00fJA8vzUJfVlCG/fRGvBKEk5YDeH430UO4='} 2023-10-17T02:51:11.184 INFO:teuthology.orchestra.console.smithi053:Power off 2023-10-17T02:51:11.185 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-17T02:51:11.209 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2023-10-17T02:51:11.210 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T02:51:11.237 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2023-10-17T02:51:15.239 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T02:51:15.348 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2023-10-17T02:51:19.350 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-17T02:51:19.462 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2023-10-17T02:51:19.563 INFO:teuthology.orchestra.console.smithi053:Power off completed 2023-10-17T02:51:19.669 INFO:teuthology.run:Summary data: description: crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} duration: 11606.329313993454 failure_reason: 'Command failed (workunit test cls/test_cls_lock.sh) on smithi053 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=06f14dfbeeb0dce341775222d82ea9644e37c496 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh''' flavor: crimson owner: scheduled_sjust@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a1891fa048a34883837698357c050253 status: fail success: false 2023-10-17T02:51:19.670 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-17T02:51:19.753 INFO:teuthology.run:FAIL