2024-04-16T05:51:15.231 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T05:51:15.231 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938 --verbose --name nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi --owner scheduled_nmordech@teuthology --description crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} -- /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938/orig.config.yaml 2024-04-16T05:51:15.262 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T05:51:15.352 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938 branch: wip-nitzan-seastore-re-enable-gate-debug-flag description: crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} email: null first_in_suite: false job_id: '7657938' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-nitzan-seastore-re-enable-gate-debug-flag ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-nitzan-seastore-re-enable-gate-debug-flag env: CRIMSON_COMPAT: '1' sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9883 sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc sleep_before_teardown: 0 suite: crimson-rados:rbd suite_branch: wip-nitzan-seastore-re-enable-gate-debug-flag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc targets: smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFdcU0XPAzrvQBbtautmoCISNbXxgmRUfXjJ9U2mrm0GFLwrq4AlHFIDFIxMsK9mt8RIXf4akKLJpLrdgG7Slh0= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - cls/test_cls_rbd.sh - cls/test_cls_lock.sh - cls/test_cls_journal.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_05:45:04 tube: smithi user: nmordech verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T05:51:15.352 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa; will attempt to use it 2024-04-16T05:51:15.353 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks 2024-04-16T05:51:15.353 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T05:51:15.355 INFO:teuthology.task.internal:Checking packages... 2024-04-16T05:51:15.378 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc' 2024-04-16T05:51:15.379 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:51:15.379 INFO:teuthology.packaging:ref: None 2024-04-16T05:51:15.379 INFO:teuthology.packaging:tag: None 2024-04-16T05:51:15.379 INFO:teuthology.packaging:branch: wip-nitzan-seastore-re-enable-gate-debug-flag 2024-04-16T05:51:15.379 INFO:teuthology.packaging:sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T05:51:15.379 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-nitzan-seastore-re-enable-gate-debug-flag 2024-04-16T05:51:15.661 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3085.gc0a28781 2024-04-16T05:51:15.662 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T05:51:15.673 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T05:51:15.673 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T05:51:15.717 INFO:teuthology.task.internal:Saving configuration 2024-04-16T05:51:15.728 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T05:51:15.748 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T05:51:15.773 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:46:05.980831', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFdcU0XPAzrvQBbtautmoCISNbXxgmRUfXjJ9U2mrm0GFLwrq4AlHFIDFIxMsK9mt8RIXf4akKLJpLrdgG7Slh0='} 2024-04-16T05:51:15.773 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T05:51:15.785 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-16T05:51:15.785 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T05:51:15.851 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe7bd5c3e50>, signals=[15]) 2024-04-16T05:51:15.851 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T05:51:15.862 INFO:teuthology.task.internal:Opening connections... 2024-04-16T05:51:15.862 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T05:51:15.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:51:15.954 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T05:51:15.965 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-04-16T05:51:15.982 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-04-16T05:51:15.982 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-04-16T05:51:16.038 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T05:51:16.039 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T05:51:16.040 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-04-16T05:51:16.062 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T05:51:16.072 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T05:51:16.101 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T05:51:16.101 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-04-16T05:51:16.117 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T05:51:16.136 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T05:51:16.137 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-04-16T05:51:16.177 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T05:51:16.215 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T05:51:16.236 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T05:51:16.236 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T05:51:16.236 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T05:51:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:51:16.252 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-16T05:51:16.307 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-04-16T05:51:16.308 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-04-16T05:51:16.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-04-16T05:51:18.501 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-16T05:51:20.168 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-04-16T05:51:21.384 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-04-16T05:51:23.394 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T05:51:35.360 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 748 kB/s | 6.2 MB 00:08 2024-04-16T05:51:36.952 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-16T05:51:44.758 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 21 MB 00:07 2024-04-16T05:51:49.830 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-16T05:51:52.204 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T05:51:52.204 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T05:51:52.367 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T05:51:52.378 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:51:52.378 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-04-16T05:51:52.379 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:51:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 77 M 2024-04-16T05:51:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-04-16T05:51:52.380 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T05:51:52.550 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.6 MB/s | 277 kB 00:00 2024-04-16T05:51:53.165 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-04-16T05:51:54.023 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-04-16T05:51:54.290 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 16 MB/s | 28 MB 00:01 2024-04-16T05:51:54.293 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:51:54.294 INFO:teuthology.orchestra.run.smithi203.stdout:Total 40 MB/s | 77 MB 00:01 2024-04-16T05:51:54.294 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T05:51:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T05:51:54.674 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T05:51:55.522 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T05:51:55.523 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T05:51:57.853 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T05:51:58.472 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T05:51:58.568 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:52:01.352 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:52:01.506 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:52:06.215 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:52:06.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:11.052 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:40.585 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:40.585 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T05:52:40.585 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:52:40.629 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:41.016 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:41.016 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T05:52:41.016 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:52:41.016 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.419 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:52:41.420 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T05:52:41.761 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T05:52:41.858 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.858 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T05:52:41.858 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T05:52:41.859 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T05:52:41.859 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T05:52:41.859 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T05:52:41.859 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T05:52:41.859 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T05:52:41.940 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:41.941 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T05:52:41.941 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-04-16T05:52:41.975 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:52:41.976 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T05:52:41.977 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T05:52:41.977 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-438.el9.x86_64 2024-04-16T05:52:41.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T05:52:42.570 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-04-16T05:52:43.913 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T05:52:43.931 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-04-16T05:52:43.934 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-16T05:52:43.949 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.u5NyfpFUSc 2024-04-16T05:52:43.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.u5NyfpFUSc 2024-04-16T05:52:44.018 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.u5NyfpFUSc 2024-04-16T05:52:44.154 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.u5NyfpFUSc is 6KB 2024-04-16T05:52:44.202 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.u5NyfpFUSc 2024-04-16T05:52:44.216 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T05:52:44.281 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T05:52:44.281 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T05:52:44.281 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T05:52:44.281 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T05:52:44.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T05:52:44.433 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-04-16T05:53:14.465 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T05:53:14.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:53:14.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:53:49.087 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-16T05:53:58.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:53:58.097 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:54:16.479 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-04-16T05:54:28.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:54:28.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:54:31.377 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-16T05:54:31.881 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:54:31.881 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T05:54:31.881 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-04-16T05:54:31.896 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-438.el9.x86_64 2024-04-16T05:54:31.896 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T05:54:31.896 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T05:54:31.896 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T05:54:32.897 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T05:54:32.897 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T05:54:32.990 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-04-16T05:54:33.024 DEBUG:teuthology.parallel:result is None 2024-04-16T05:54:33.024 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T05:54:33.033 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T05:54:33.033 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T05:54:33.049 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T05:54:33.117 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T05:54:33.148 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T05:54:33.148 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T05:54:33.172 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T05:54:33.183 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T05:54:33.183 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T05:54:33.245 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:54:33.259 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:54:33.260 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T05:54:33.269 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T05:54:33.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T05:54:33.331 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T05:54:33.340 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T05:54:33.340 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T05:54:33.386 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:54:33.471 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:54:33.525 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:54:33.610 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:54:33.653 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T05:54:33.654 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T05:54:33.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-04-16T05:54:33.819 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T05:54:34.326 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T05:54:34.335 INFO:teuthology.task.internal:Starting timer... 2024-04-16T05:54:34.335 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T05:54:34.378 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T05:54:34.408 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T05:54:34.409 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-04-16T05:54:34.458 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-04-16T05:54:34.463 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T05:54:34.463 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-04-16T05:54:34.478 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-04-16T05:54:34.479 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-04-16T05:54:34.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T05:54:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:54:34.549 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T05:54:34.550 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-04-16T05:54:34.619 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T05:54:34.629 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T05:54:34.646 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T05:54:34.647 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T05:58:51.872 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-04-16T05:58:51.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:58:51.875 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:58:51.966 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-04-16T05:58:52.041 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-04-16T05:58:52.042 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T05:58:52.053 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T05:58:52.053 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T05:58:52.054 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T05:58:52.114 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T05:58:52.131 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T05:58:52.162 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-04-16T05:58:52.176 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-04-16T05:58:52.194 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T05:58:52.211 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T05:58:52.268 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-16T05:58:52.271 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T05:58:52.271 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-16T05:58:52.271 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:52.271 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:52.272 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:52.272 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:52.272 INFO:teuthology.run_tasks:Running task install... 2024-04-16T05:58:52.281 DEBUG:teuthology.task.install:project ceph 2024-04-16T05:58:52.281 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}} 2024-04-16T05:58:52.281 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'} 2024-04-16T05:58:52.281 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T05:58:52.287 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T05:58:52.287 INFO:teuthology.task.install:extra packages: [] 2024-04-16T05:58:52.287 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'tag': None, 'wait_for_package': False} 2024-04-16T05:58:52.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T05:58:52.432 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/ 2024-04-16T05:58:52.432 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3085.gc0a28781 2024-04-16T05:58:52.544 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T05:58:52.544 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T05:58:52.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T05:58:52.573 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T05:58:52.573 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T05:58:52.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T05:58:52.726 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T05:58:52.795 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-04-16T05:58:52.796 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-04-16T05:58:53.039 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-04-16T05:58:53.066 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-04-16T05:58:53.693 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 273 kB/s | 87 kB 00:00 2024-04-16T05:58:54.074 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-16T05:58:54.510 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 5.2 kB/s | 2.0 kB 00:00 2024-04-16T05:58:54.902 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 19 MB/s | 6.7 MB 00:00 2024-04-16T05:58:56.336 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-04-16T05:58:57.395 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-16T05:58:59.394 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-16T05:59:03.645 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.2 MB 00:00 2024-04-16T05:59:05.471 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-16T05:59:26.023 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-04-16T05:59:31.178 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-16T05:59:34.009 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 10 M 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.4 M 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.7 M 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T05:59:34.021 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.9 M 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 19 M 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 25 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 796 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 549 k 2024-04-16T05:59:34.022 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.2 M 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 45 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 135 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 159 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 315 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 294 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 99 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T05:59:34.023 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout:Install 30 Packages 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-04-16T05:59:34.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:59:34.025 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 59 M 2024-04-16T05:59:34.025 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T05:59:36.837 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-base-19.0.0-3085.gc0a28781.el9.x86 2.2 MB/s | 5.9 MB 00:02 2024-04-16T05:59:36.921 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-selinux-19.0.0-3085.gc0a28781.el9. 302 kB/s | 25 kB 00:00 2024-04-16T05:59:37.213 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): libcephfs2-19.0.0-3085.gc0a28781.el9.x8 2.7 MB/s | 796 kB 00:00 2024-04-16T05:59:37.438 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): libradosstriper1-19.0.0-3085.gc0a28781. 2.4 MB/s | 549 kB 00:00 2024-04-16T05:59:38.055 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): ceph-radosgw-19.0.0-3085.gc0a28781.el9. 2.6 MB/s | 10 MB 00:03 2024-04-16T05:59:38.122 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-ceph-argparse-19.0.0-3085.gc0a2 679 kB/s | 45 kB 00:00 2024-04-16T05:59:38.180 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-ceph-common-19.0.0-3085.gc0a287 2.3 MB/s | 135 kB 00:00 2024-04-16T05:59:38.247 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-cephfs-19.0.0-3085.gc0a28781.el 2.3 MB/s | 159 kB 00:00 2024-04-16T05:59:38.364 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-rados-19.0.0-3085.gc0a28781.el9 2.6 MB/s | 315 kB 00:00 2024-04-16T05:59:38.465 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): python3-rbd-19.0.0-3085.gc0a28781.el9. 2.9 MB/s | 294 kB 00:00 2024-04-16T05:59:38.549 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): librgw2-19.0.0-3085.gc0a28781.el9.x86_ 4.7 MB/s | 5.2 MB 00:01 2024-04-16T05:59:38.573 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): python3-rgw-19.0.0-3085.gc0a28781.el9. 914 kB/s | 99 kB 00:00 2024-04-16T05:59:38.691 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2024-04-16T05:59:38.716 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-04-16T05:59:38.741 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-04-16T05:59:38.791 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-04-16T05:59:38.817 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-04-16T05:59:38.842 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 830 kB/s | 41 kB 00:00 2024-04-16T05:59:38.867 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-16T05:59:38.892 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 696 kB/s | 35 kB 00:00 2024-04-16T05:59:38.918 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-04-16T05:59:39.026 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.3 MB/s | 309 kB 00:00 2024-04-16T05:59:39.060 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 780 kB/s | 26 kB 00:00 2024-04-16T05:59:39.094 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-16T05:59:39.161 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-04-16T05:59:39.186 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-16T05:59:39.228 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-04-16T05:59:39.253 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-04-16T05:59:39.295 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-16T05:59:39.937 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): librados2-19.0.0-3085.gc0a28781.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-04-16T05:59:40.096 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): librbd1-19.0.0-3085.gc0a28781.el9.x86_ 4.6 MB/s | 3.7 MB 00:00 2024-04-16T05:59:40.946 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): ceph-common-19.0.0-3085.gc0a28781.el9. 2.9 MB/s | 19 MB 00:06 2024-04-16T05:59:40.950 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:59:40.951 INFO:teuthology.orchestra.run.smithi203.stdout:Total 8.5 MB/s | 59 MB 00:06 2024-04-16T05:59:40.951 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T05:59:40.989 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T05:59:40.989 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T05:59:41.466 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T05:59:41.466 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T05:59:42.971 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T05:59:43.456 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T05:59:43.498 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T05:59:43.590 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T05:59:43.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 3/34 2024-04-16T05:59:43.699 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T05:59:43.745 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T05:59:43.790 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T05:59:43.820 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T05:59:43.846 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T05:59:43.869 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T05:59:43.885 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T05:59:44.078 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T05:59:44.109 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T05:59:44.462 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T05:59:44.497 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T05:59:44.564 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T05:59:44.611 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9. 12/34 2024-04-16T05:59:44.671 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 13/34 2024-04-16T05:59:44.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 14/34 2024-04-16T05:59:44.784 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T05:59:44.808 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 16/34 2024-04-16T05:59:44.883 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 16/34 2024-04-16T05:59:44.924 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T05:59:45.034 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T05:59:45.066 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T05:59:45.107 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T05:59:45.139 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T05:59:45.168 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T05:59:45.200 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T05:59:45.241 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T05:59:45.807 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T05:59:45.840 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 26/34 2024-04-16T05:59:45.910 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 26/34 2024-04-16T05:59:45.979 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 27/34 2024-04-16T05:59:46.008 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x8 28/34 2024-04-16T05:59:48.407 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:48.455 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:49.138 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:49.183 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 30/34 2024-04-16T05:59:49.293 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 30/34 2024-04-16T05:59:49.293 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T05:59:49.293 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T05:59:49.294 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T05:59:49.326 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T05:59:59.356 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-04-16T05:59:59.357 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:00.431 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 32/34 2024-04-16T06:00:00.475 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 32/34 2024-04-16T06:00:00.475 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:00.475 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T06:00:00.475 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T06:00:00.475 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T06:00:00.476 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:00.477 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T06:00:00.567 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T06:00:00.567 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T06:00:01.795 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 1/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 3/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 5/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 6/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 7/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9. 8/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x8 9/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 10/34 2024-04-16T06:00:01.796 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 12/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 13/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T06:00:01.797 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T06:00:01.798 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 33/34 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T06:00:02.883 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:02.884 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:00:03.217 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-04-16T06:00:03.803 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:32 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:00:03.966 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:00:03.967 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:00:03.967 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 84 M 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 174 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T06:00:03.968 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout:Install 11 Packages 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:03.969 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 87 M 2024-04-16T06:00:03.970 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 814 M 2024-04-16T06:00:03.970 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:00:04.987 INFO:teuthology.orchestra.run.smithi203.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-04-16T06:00:05.104 INFO:teuthology.orchestra.run.smithi203.stdout:(2/11): libcephsqlite-19.0.0-3085.gc0a28781.el9 614 kB/s | 174 kB 00:00 2024-04-16T06:00:05.137 INFO:teuthology.orchestra.run.smithi203.stdout:(3/11): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-04-16T06:00:05.163 INFO:teuthology.orchestra.run.smithi203.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-04-16T06:00:05.179 INFO:teuthology.orchestra.run.smithi203.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-04-16T06:00:05.205 INFO:teuthology.orchestra.run.smithi203.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-04-16T06:00:05.297 INFO:teuthology.orchestra.run.smithi203.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 5.0 MB/s | 469 kB 00:00 2024-04-16T06:00:05.330 INFO:teuthology.orchestra.run.smithi203.stdout:(8/11): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-04-16T06:00:05.389 INFO:teuthology.orchestra.run.smithi203.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-04-16T06:00:05.939 INFO:teuthology.orchestra.run.smithi203.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 200 kB/s | 122 kB 00:00 2024-04-16T06:00:09.656 INFO:teuthology.orchestra.run.smithi203.stdout:(11/11): ceph-test-19.0.0-3085.gc0a28781.el9.x8 17 MB/s | 84 MB 00:04 2024-04-16T06:00:09.660 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:09.660 INFO:teuthology.orchestra.run.smithi203.stdout:Total 15 MB/s | 87 MB 00:05 2024-04-16T06:00:09.702 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:00:09.726 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:00:09.726 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:00:10.364 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:00:10.364 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:00:11.366 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:00:11.427 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T06:00:11.457 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T06:00:11.483 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T06:00:11.551 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T06:00:11.583 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T06:00:11.627 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T06:00:11.672 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T06:00:11.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T06:00:11.735 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T06:00:11.784 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T06:00:11.814 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 10/11 2024-04-16T06:00:23.061 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 10/11 2024-04-16T06:00:23.147 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 11/11 2024-04-16T06:00:29.545 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 11/11 2024-04-16T06:00:29.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 1/11 2024-04-16T06:00:29.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 2/11 2024-04-16T06:00:29.545 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T06:00:29.546 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T06:00:29.952 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T06:00:29.953 INFO:teuthology.orchestra.run.smithi203.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T06:00:29.954 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:29.954 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:00:30.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-04-16T06:00:30.839 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:59 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:00:30.991 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 6.5 k 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 2.2 M 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 1.6 M 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 249 k 2024-04-16T06:00:30.994 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.0 M 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 71 M 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T06:00:30.995 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T06:00:30.996 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout:Install 36 Packages 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:30.997 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 84 M 2024-04-16T06:00:30.998 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 922 M 2024-04-16T06:00:30.998 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:00:31.523 INFO:teuthology.orchestra.run.smithi203.stdout:(1/36): ceph-19.0.0-3085.gc0a28781.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-04-16T06:00:31.657 INFO:teuthology.orchestra.run.smithi203.stdout:(2/36): ceph-mds-19.0.0-3085.gc0a28781.el9.x86_ 5.9 MB/s | 2.2 MB 00:00 2024-04-16T06:00:31.999 INFO:teuthology.orchestra.run.smithi203.stdout:(3/36): ceph-mgr-19.0.0-3085.gc0a28781.el9.x86_ 2.2 MB/s | 1.6 MB 00:00 2024-04-16T06:00:32.108 INFO:teuthology.orchestra.run.smithi203.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3085.gc0a2 2.2 MB/s | 249 kB 00:00 2024-04-16T06:00:32.225 INFO:teuthology.orchestra.run.smithi203.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T06:00:32.267 INFO:teuthology.orchestra.run.smithi203.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T06:00:32.292 INFO:teuthology.orchestra.run.smithi203.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-04-16T06:00:32.326 INFO:teuthology.orchestra.run.smithi203.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-04-16T06:00:32.359 INFO:teuthology.orchestra.run.smithi203.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T06:00:32.401 INFO:teuthology.orchestra.run.smithi203.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-04-16T06:00:32.469 INFO:teuthology.orchestra.run.smithi203.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-04-16T06:00:32.510 INFO:teuthology.orchestra.run.smithi203.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-04-16T06:00:32.544 INFO:teuthology.orchestra.run.smithi203.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-04-16T06:00:32.578 INFO:teuthology.orchestra.run.smithi203.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-04-16T06:00:32.653 INFO:teuthology.orchestra.run.smithi203.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-04-16T06:00:32.912 INFO:teuthology.orchestra.run.smithi203.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-04-16T06:00:33.129 INFO:teuthology.orchestra.run.smithi203.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 698 kB/s | 151 kB 00:00 2024-04-16T06:00:33.187 INFO:teuthology.orchestra.run.smithi203.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 745 kB/s | 43 kB 00:00 2024-04-16T06:00:33.271 INFO:teuthology.orchestra.run.smithi203.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-16T06:00:33.371 INFO:teuthology.orchestra.run.smithi203.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-04-16T06:00:33.422 INFO:teuthology.orchestra.run.smithi203.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-04-16T06:00:33.472 INFO:teuthology.orchestra.run.smithi203.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-04-16T06:00:33.522 INFO:teuthology.orchestra.run.smithi203.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-04-16T06:00:33.581 INFO:teuthology.orchestra.run.smithi203.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-04-16T06:00:33.631 INFO:teuthology.orchestra.run.smithi203.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-04-16T06:00:33.690 INFO:teuthology.orchestra.run.smithi203.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 792 kB/s | 46 kB 00:00 2024-04-16T06:00:33.740 INFO:teuthology.orchestra.run.smithi203.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-04-16T06:00:33.807 INFO:teuthology.orchestra.run.smithi203.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-16T06:00:33.866 INFO:teuthology.orchestra.run.smithi203.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-04-16T06:00:33.942 INFO:teuthology.orchestra.run.smithi203.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-04-16T06:00:34.000 INFO:teuthology.orchestra.run.smithi203.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-04-16T06:00:34.067 INFO:teuthology.orchestra.run.smithi203.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-04-16T06:00:34.168 INFO:teuthology.orchestra.run.smithi203.stdout:(33/36): ceph-mon-19.0.0-3085.gc0a28781.el9.x86 1.9 MB/s | 5.0 MB 00:02 2024-04-16T06:00:34.201 INFO:teuthology.orchestra.run.smithi203.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-04-16T06:00:34.293 INFO:teuthology.orchestra.run.smithi203.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-04-16T06:00:35.127 INFO:teuthology.orchestra.run.smithi203.stdout:(36/36): ceph-osd-19.0.0-3085.gc0a28781.el9.x86 20 MB/s | 71 MB 00:03 2024-04-16T06:00:35.131 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:35.131 INFO:teuthology.orchestra.run.smithi203.stdout:Total 20 MB/s | 84 MB 00:04 2024-04-16T06:00:35.157 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:00:35.200 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:00:35.201 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:00:35.938 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:00:35.939 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:00:37.534 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:00:37.596 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T06:00:37.629 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T06:00:37.665 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T06:00:37.735 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T06:00:37.772 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T06:00:37.805 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T06:00:37.897 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T06:00:37.975 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T06:00:38.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T06:00:38.088 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T06:00:38.142 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T06:00:38.181 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T06:00:38.230 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T06:00:38.284 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T06:00:38.470 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T06:00:38.519 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T06:00:38.556 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T06:00:38.592 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T06:00:38.623 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T06:00:38.699 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T06:00:38.733 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T06:00:38.766 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T06:00:38.807 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T06:00:38.842 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T06:00:38.885 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T06:00:38.974 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T06:00:39.015 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T06:00:39.088 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T06:00:39.226 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T06:00:39.351 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T06:00:39.568 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9. 31/36 2024-04-16T06:00:39.596 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 32/36 2024-04-16T06:00:39.623 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 32/36 2024-04-16T06:00:39.623 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:39.623 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T06:00:39.623 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T06:00:39.624 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T06:00:39.624 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:50.463 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 33/36 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 33/36 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T06:00:50.491 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:51.039 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 34/36 2024-04-16T06:00:51.063 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 34/36 2024-04-16T06:00:51.063 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:51.063 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T06:00:51.064 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T06:00:51.064 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T06:00:51.064 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:51.298 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 35/36 2024-04-16T06:00:51.322 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 35/36 2024-04-16T06:00:51.322 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:51.322 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T06:00:51.323 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T06:00:51.323 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T06:00:51.323 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:51.388 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 36/36 2024-04-16T06:00:58.443 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 36/36 2024-04-16T06:00:58.443 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 1/36 2024-04-16T06:00:58.443 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 2/36 2024-04-16T06:00:58.443 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 3/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 4/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 5/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9. 6/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T06:00:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T06:00:58.445 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T06:00:58.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T06:00:58.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T06:00:58.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T06:00:58.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:59.456 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T06:00:59.457 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T06:00:59.458 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:00:59.459 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:00:59.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-04-16T06:01:00.365 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:29 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:00.469 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:00.524 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:00.524 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:00.525 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:00.611 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-04-16T06:01:01.204 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:30 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:01.356 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:01.356 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 762 k 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 762 k 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 768 k 2024-04-16T06:01:01.357 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:01.635 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.0.0-3085.gc0a28781.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-04-16T06:01:01.635 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:01.636 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-04-16T06:01:01.636 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:01.641 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:01.641 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:01.647 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:01.647 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:01.718 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:02.178 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:01:02.272 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:01:02.692 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:01:02.969 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:01:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:02.970 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:02.970 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:02.970 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:03.191 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T06:01:03.777 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:32 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:03.931 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 157 k 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:03.932 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:03.933 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 157 k 2024-04-16T06:01:03.933 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 640 k 2024-04-16T06:01:03.933 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:04.159 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.0.0-3085.gc0a287 694 kB/s | 157 kB 00:00 2024-04-16T06:01:04.160 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:04.160 INFO:teuthology.orchestra.run.smithi203.stdout:Total 688 kB/s | 157 kB 00:00 2024-04-16T06:01:04.160 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:04.169 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:04.170 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:04.227 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:04.227 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:04.378 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:04.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:01:04.461 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:01:04.461 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:01:04.461 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T06:01:04.461 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:05.442 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:01:05.443 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:05.443 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:05.443 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:05.443 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:05.443 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:05.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-04-16T06:01:06.276 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:35 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:06.376 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:06.428 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:06.429 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:06.429 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:06.508 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T06:01:07.090 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:07.243 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 3.5 M 2024-04-16T06:01:07.244 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 34 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 16 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout:Install 11 Packages 2024-04-16T06:01:07.245 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:07.246 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 7.6 M 2024-04-16T06:01:07.246 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 100 M 2024-04-16T06:01:07.246 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:09.839 INFO:teuthology.orchestra.run.smithi203.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3085.gc0a 54 kB/s | 16 kB 00:00 2024-04-16T06:01:09.864 INFO:teuthology.orchestra.run.smithi203.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3085.gc0 103 kB/s | 34 kB 00:00 2024-04-16T06:01:09.998 INFO:teuthology.orchestra.run.smithi203.stdout:(3/11): ceph-mgr-dashboard-19.0.0-3085.gc0a2878 7.6 MB/s | 3.5 MB 00:00 2024-04-16T06:01:10.057 INFO:teuthology.orchestra.run.smithi203.stdout:(4/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 238 kB 00:00 2024-04-16T06:01:10.174 INFO:teuthology.orchestra.run.smithi203.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-04-16T06:01:10.199 INFO:teuthology.orchestra.run.smithi203.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 137 kB/s | 19 kB 00:00 2024-04-16T06:01:10.324 INFO:teuthology.orchestra.run.smithi203.stdout:(7/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-04-16T06:01:10.375 INFO:teuthology.orchestra.run.smithi203.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 615 kB/s | 31 kB 00:00 2024-04-16T06:01:10.408 INFO:teuthology.orchestra.run.smithi203.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 692 kB/s | 144 kB 00:00 2024-04-16T06:01:10.450 INFO:teuthology.orchestra.run.smithi203.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-04-16T06:01:10.500 INFO:teuthology.orchestra.run.smithi203.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.2 MB/s | 2.0 MB 00:00 2024-04-16T06:01:10.504 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:10.504 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.3 MB/s | 7.6 MB 00:03 2024-04-16T06:01:10.524 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:10.541 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:10.542 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:10.723 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:10.723 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:11.213 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:11.299 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T06:01:11.337 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T06:01:11.437 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T06:01:11.572 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T06:01:11.633 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T06:01:11.716 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T06:01:11.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T06:01:11.788 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T06:01:11.821 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9 9/11 2024-04-16T06:01:12.878 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el 10/11 2024-04-16T06:01:12.922 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 11/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 11/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el 1/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 2/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9 3/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T06:01:14.253 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T06:01:14.587 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T06:01:14.587 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:14.587 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:14.587 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T06:01:14.587 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:14.588 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:14.841 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T06:01:15.422 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:15.575 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 7.4 M 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:15.576 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T06:01:15.577 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-04-16T06:01:15.578 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-04-16T06:01:15.579 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:17.635 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-04-16T06:01:17.668 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 543 kB/s | 17 kB 00:00 2024-04-16T06:01:17.710 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 34 kB 00:00 2024-04-16T06:01:17.761 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-04-16T06:01:17.936 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-04-16T06:01:18.003 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-04-16T06:01:18.028 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-04-16T06:01:18.337 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-04-16T06:01:18.479 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-16T06:01:18.504 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-04-16T06:01:18.755 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-3 5.9 MB/s | 7.4 MB 00:01 2024-04-16T06:01:18.971 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 499 kB/s | 245 kB 00:00 2024-04-16T06:01:18.973 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:18.973 INFO:teuthology.orchestra.run.smithi203.stdout:Total 11 MB/s | 37 MB 00:03 2024-04-16T06:01:19.016 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:19.040 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:19.040 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:19.390 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:19.390 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:20.095 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:20.169 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T06:01:20.201 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T06:01:20.294 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T06:01:20.329 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T06:01:20.515 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T06:01:20.545 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T06:01:20.621 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T06:01:21.267 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T06:01:21.384 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T06:01:22.650 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T06:01:23.687 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T06:01:23.715 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 12/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 12/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 1/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T06:01:26.463 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T06:01:26.464 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T06:01:26.464 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T06:01:26.464 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T06:01:26.464 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T06:01:26.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T06:01:26.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T06:01:26.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T06:01:26.886 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:26.886 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:27.328 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-04-16T06:01:27.922 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:56 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:28.075 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 49 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T06:01:28.077 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-04-16T06:01:28.078 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:28.079 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-04-16T06:01:28.079 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-04-16T06:01:28.079 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:28.576 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 206 kB/s | 24 kB 00:00 2024-04-16T06:01:28.601 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 125 kB/s | 18 kB 00:00 2024-04-16T06:01:28.660 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-04-16T06:01:28.685 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-04-16T06:01:28.710 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T06:01:28.738 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-04-16T06:01:28.761 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): ceph-mgr-rook-19.0.0-3085.gc0a28781.el9 162 kB/s | 49 kB 00:00 2024-04-16T06:01:28.803 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 224 kB/s | 15 kB 00:00 2024-04-16T06:01:28.828 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 274 kB/s | 32 kB 00:00 2024-04-16T06:01:28.878 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-16T06:01:28.904 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-04-16T06:01:28.929 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-04-16T06:01:28.962 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-04-16T06:01:28.964 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:28.964 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-04-16T06:01:28.965 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:28.978 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:28.978 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:29.068 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:29.070 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:29.578 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:29.736 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T06:01:29.797 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T06:01:29.861 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T06:01:29.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T06:01:29.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T06:01:30.026 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T06:01:30.143 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T06:01:30.191 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T06:01:30.922 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T06:01:30.973 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T06:01:31.007 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T06:01:31.055 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T06:01:31.077 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 13/13 2024-04-16T06:01:31.953 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 13/13 2024-04-16T06:01:31.953 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 1/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T06:01:31.954 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T06:01:32.328 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:32.329 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:32.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T06:01:33.192 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:02 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:33.348 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 142 k 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:33.349 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-04-16T06:01:33.350 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-04-16T06:01:33.350 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:33.350 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-04-16T06:01:33.350 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-04-16T06:01:33.350 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:33.976 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-16T06:01:34.076 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-04-16T06:01:34.101 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3085.gc0a28781.e 481 kB/s | 142 kB 00:00 2024-04-16T06:01:34.202 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 461 kB/s | 58 kB 00:00 2024-04-16T06:01:34.227 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-04-16T06:01:34.486 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-04-16T06:01:34.488 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:34.488 INFO:teuthology.orchestra.run.smithi203.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-04-16T06:01:34.502 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:34.513 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:34.513 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:34.604 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:34.607 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:34.899 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:35.077 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T06:01:35.121 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T06:01:35.637 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T06:01:35.723 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T06:01:35.787 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T06:01:35.817 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 6/6 2024-04-16T06:01:36.683 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 6/6 2024-04-16T06:01:36.684 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/6 2024-04-16T06:01:36.684 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T06:01:36.684 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T06:01:36.684 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T06:01:36.684 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T06:01:36.979 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:36.980 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:37.250 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-04-16T06:01:37.834 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:06 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:37.988 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:37.988 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:37.988 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 870 k 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-16T06:01:37.989 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:37.990 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 949 k 2024-04-16T06:01:37.990 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 4.0 M 2024-04-16T06:01:37.990 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:38.122 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 598 kB/s | 79 kB 00:00 2024-04-16T06:01:38.439 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.0.0-3085.gc0a28781.el9.x86_ 1.9 MB/s | 870 kB 00:00 2024-04-16T06:01:38.440 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:38.440 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.1 MB/s | 949 kB 00:00 2024-04-16T06:01:38.440 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:38.462 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:38.462 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:38.535 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:38.535 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:38.712 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:38.865 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T06:01:38.949 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2/2 2024-04-16T06:01:39.654 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2/2 2024-04-16T06:01:39.654 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/2 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:39.963 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:40.226 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-04-16T06:01:40.822 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:09 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:40.975 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 264 k 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-04-16T06:01:40.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:40.977 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 332 k 2024-04-16T06:01:40.977 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.5 M 2024-04-16T06:01:40.977 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:41.120 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 482 kB/s | 69 kB 00:00 2024-04-16T06:01:41.362 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-volume-19.0.0-3085.gc0a28781.el9.no 686 kB/s | 264 kB 00:00 2024-04-16T06:01:41.363 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:41.363 INFO:teuthology.orchestra.run.smithi203.stdout:Total 859 kB/s | 332 kB 00:00 2024-04-16T06:01:41.363 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:41.372 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:41.372 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:41.413 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:41.414 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:41.649 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:41.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T06:01:41.898 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2/2 2024-04-16T06:01:41.915 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2/2 2024-04-16T06:01:41.915 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:01:41.915 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T06:01:41.915 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:42.663 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 1/2 2024-04-16T06:01:43.072 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T06:01:43.072 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:43.072 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:43.072 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:43.073 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T06:01:43.073 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:43.073 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:43.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-04-16T06:01:43.987 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:12 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:44.140 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 125 k 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:44.141 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 125 k 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 445 k 2024-04-16T06:01:44.142 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:44.461 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.0.0-3085.gc0a28781.el9.x86_64 391 kB/s | 125 kB 00:00 2024-04-16T06:01:44.462 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:44.462 INFO:teuthology.orchestra.run.smithi203.stdout:Total 389 kB/s | 125 kB 00:00 2024-04-16T06:01:44.462 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:44.467 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:44.467 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:44.523 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:44.523 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:44.669 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:44.777 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.272 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:45.579 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:45.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-04-16T06:01:46.424 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:15 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:46.525 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:46.579 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:46.580 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:46.580 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:46.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-04-16T06:01:47.244 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:16 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:47.395 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 32 k 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:47.396 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:47.397 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:47.397 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:47.397 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 32 k 2024-04-16T06:01:47.397 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 142 k 2024-04-16T06:01:47.397 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:47.807 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.0.0-3085.gc0a28781.el9.x86_6 77 kB/s | 32 kB 00:00 2024-04-16T06:01:47.808 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:47.808 INFO:teuthology.orchestra.run.smithi203.stdout:Total 77 kB/s | 32 kB 00:00 2024-04-16T06:01:47.808 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:47.812 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:47.812 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:47.840 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:47.840 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:47.933 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:48.038 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:48.425 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:48.685 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:48.685 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:48.685 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:48.686 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:48.686 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:48.686 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:48.881 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-04-16T06:01:49.468 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:18 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:49.567 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:49.619 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:49.620 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:49.620 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:49.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-04-16T06:01:50.282 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:19 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:50.382 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:50.434 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:50.435 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:50.435 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:50.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-04-16T06:01:51.090 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:20 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:51.189 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:51.240 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:51.242 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:51.242 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:51.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-04-16T06:01:51.901 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:20 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:51.999 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:52.051 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:52.052 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:52.052 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:52.133 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-04-16T06:01:52.717 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:21 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:52.815 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:52.869 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:52.870 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:52.870 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:52.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-04-16T06:01:53.549 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:22 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:53.658 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:53.710 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:53.712 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-04-16T06:01:53.712 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:53.790 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-04-16T06:01:54.374 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:23 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 90 k 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:54.525 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 90 k 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 313 k 2024-04-16T06:01:54.526 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:54.836 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.0.0-3085.gc0a28781.el9.x86_64.rpm 290 kB/s | 90 kB 00:00 2024-04-16T06:01:54.836 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:54.836 INFO:teuthology.orchestra.run.smithi203.stdout:Total 289 kB/s | 90 kB 00:00 2024-04-16T06:01:54.837 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:54.845 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:54.845 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:54.900 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:54.901 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:55.055 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:55.160 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:55.537 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:55.843 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:56.113 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-04-16T06:01:56.694 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:25 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:01:56.849 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:01:56.849 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.4 M 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.4 M 2024-04-16T06:01:56.850 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 29 M 2024-04-16T06:01:56.851 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:01:57.449 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.0.0-3085.gc0a28781.el9.x86_64.rpm 5.7 MB/s | 3.4 MB 00:00 2024-04-16T06:01:57.449 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:57.449 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.7 MB/s | 3.4 MB 00:00 2024-04-16T06:01:57.450 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:01:57.460 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:01:57.461 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:01:57.534 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:01:57.535 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:01:58.088 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:01:58.158 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T06:01:58.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:01:59.452 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:01:59.696 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-04-16T06:02:00.292 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:29 ago on Tue 16 Apr 2024 05:59:31 AM UTC. 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-04-16T06:02:00.444 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 175 k 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 175 k 2024-04-16T06:02:00.445 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 673 k 2024-04-16T06:02:00.446 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-04-16T06:02:00.654 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.0.0-3085.gc0a28781.el9.x86_64.rpm 840 kB/s | 175 kB 00:00 2024-04-16T06:02:00.655 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:02:00.655 INFO:teuthology.orchestra.run.smithi203.stdout:Total 834 kB/s | 175 kB 00:00 2024-04-16T06:02:00.655 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-04-16T06:02:00.665 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-04-16T06:02:00.665 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-04-16T06:02:00.721 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-04-16T06:02:00.721 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-04-16T06:02:00.863 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-04-16T06:02:00.978 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:02:01.401 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:02:01.676 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:02:01.676 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:01.676 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-04-16T06:02:01.676 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:02:01.676 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:01.677 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-04-16T06:02:01.926 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:01.927 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:02:02.153 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T06:02:02.174 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-3085.gc0a28781.el9 2024-04-16T06:02:02.175 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3085.gc0a28781.el9 2024-04-16T06:02:02.175 INFO:teuthology.task.install:The correct ceph version 19.0.0-3085.gc0a28781 is installed. 2024-04-16T06:02:02.176 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T06:02:02.177 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:02.177 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T06:02:02.244 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T06:02:02.244 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:02.244 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T06:02:02.314 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T06:02:02.384 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T06:02:02.384 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:02.384 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T06:02:02.453 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T06:02:02.521 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T06:02:02.521 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:02.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T06:02:02.589 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T06:02:02.658 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T06:02:02.734 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T06:02:02.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-04-16T06:02:02.762 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T06:02:02.763 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T06:02:02.830 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T06:02:02.830 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T06:02:02.906 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T06:02:02.906 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T06:02:02.906 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938', 'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'description': 'crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests}', 'email': None, 'first_in_suite': False, 'job_id': '7657938', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}}, 'owner': 'scheduled_nmordech@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9883, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'sleep_before_teardown': 0, 'suite': 'crimson-rados:rbd', 'suite_branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'targets': {'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFdcU0XPAzrvQBbtautmoCISNbXxgmRUfXjJ9U2mrm0GFLwrq4AlHFIDFIxMsK9mt8RIXf4akKLJpLrdgG7Slh0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_05:45:04', 'tube': 'smithi', 'user': 'nmordech', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T06:02:02.906 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T06:02:02.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T06:02:03.024 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:03.024 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T06:02:03.080 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T06:02:03.080 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-04-16T06:02:03.135 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-16 06:01:58.434960585 +0000 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-16 05:58:29.390273597 +0000 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-16 05:58:29.390273597 +0000 2024-04-16T06:02:03.136 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-16 05:58:29.390273597 +0000 2024-04-16T06:02:03.136 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T06:02:03.203 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-16T06:02:03.203 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-16T06:02:03.203 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000206297 s, 2.5 MB/s 2024-04-16T06:02:03.204 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T06:02:03.261 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 678 Links: 1 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-16 06:01:58.434960585 +0000 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-16 05:58:29.738265099 +0000 2024-04-16T06:02:03.317 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-16 05:58:29.738265099 +0000 2024-04-16T06:02:03.318 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-16 05:58:29.738265099 +0000 2024-04-16T06:02:03.318 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T06:02:03.384 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-16T06:02:03.384 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-16T06:02:03.384 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000209286 s, 2.4 MB/s 2024-04-16T06:02:03.385 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T06:02:03.442 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-16 06:01:58.435960560 +0000 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-16 05:58:30.081256724 +0000 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-16 05:58:30.081256724 +0000 2024-04-16T06:02:03.497 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-16 05:58:30.081256724 +0000 2024-04-16T06:02:03.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T06:02:03.563 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-16T06:02:03.563 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-16T06:02:03.564 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000194826 s, 2.6 MB/s 2024-04-16T06:02:03.565 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T06:02:03.621 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-04-16T06:02:03.676 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T06:02:03.676 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:02:03.676 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-04-16T06:02:03.676 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:02:03.676 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:02:03.677 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-04-16 06:01:58.435960560 +0000 2024-04-16T06:02:03.677 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-04-16 05:58:30.432248153 +0000 2024-04-16T06:02:03.677 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-04-16 05:58:30.432248153 +0000 2024-04-16T06:02:03.677 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-04-16 05:58:30.432248153 +0000 2024-04-16T06:02:03.677 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T06:02:03.743 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-04-16T06:02:03.743 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-04-16T06:02:03.743 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000286571 s, 1.8 MB/s 2024-04-16T06:02:03.745 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T06:02:03.801 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:02:03.801 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T06:02:03.801 INFO:tasks.ceph:Generating config... 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T06:02:03.802 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T06:02:03.803 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T06:02:03.803 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T06:02:03.803 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T06:02:03.803 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T06:02:03.803 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T06:02:03.804 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T06:02:03.804 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T06:02:03.804 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T06:02:03.804 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-04-16T06:02:03.804 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-04-16T06:02:03.804 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-04-16T06:02:03.804 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T06:02:03.804 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T06:02:03.919 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T06:02:03.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T06:02:03.976 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T06:02:04.004 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.203')] 2024-04-16T06:02:04.005 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.203', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-04-16T06:02:04.005 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:04.005 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T06:02:04.060 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.203 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:04.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:04.152+0000 7f4481434f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:04.151 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:04.152+0000 7f4481434f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:04.153+0000 7f4481434f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: generated fsid 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:setting min_mon_release = quincy 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:epoch 0 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:fsid 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:last_changed 2024-04-16T06:02:04.155119+0000 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:created 2024-04-16T06:02:04.155119+0000 2024-04-16T06:02:04.152 INFO:teuthology.orchestra.run.smithi203.stdout:min_mon_release 17 (quincy) 2024-04-16T06:02:04.153 INFO:teuthology.orchestra.run.smithi203.stdout:election_strategy: 1 2024-04-16T06:02:04.153 INFO:teuthology.orchestra.run.smithi203.stdout:0: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.a 2024-04-16T06:02:04.153 INFO:teuthology.orchestra.run.smithi203.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T06:02:04.154 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T06:02:04.168 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8af020fd-9691-42e7-ac58-13efaaf4e615... 2024-04-16T06:02:04.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-04-16T06:02:04.261 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T06:02:04.262 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-04-16T06:02:04.263 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = 172.21.15.203 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: ms client mode = crc 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: ms cluster mode = crc 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon client mode = crc 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon cluster mode = crc 2024-04-16T06:02:04.264 INFO:teuthology.orchestra.run.smithi203.stdout: ms mon service mode = crc 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: ms service mode = crc 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-04-16T06:02:04.265 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-04-16T06:02:04.266 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: crimson osd obc lru size = 10 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-04-16T06:02:04.267 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T06:02:04.268 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default crimson = True 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-04-16T06:02:04.269 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 0-1 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 2-3 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-04-16T06:02:04.270 INFO:teuthology.orchestra.run.smithi203.stdout: crimson seastar cpu cores = 4-5 2024-04-16T06:02:04.276 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T06:02:04.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T06:02:04.384 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T06:02:04.384 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:04.384 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T06:02:04.400 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:04.400 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T06:02:04.456 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T06:02:04.456 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:04.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T06:02:04.456 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T06:02:04.537 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:04.537 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:04.592 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T06:02:04.593 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T06:02:04.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T06:02:04.698 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T06:02:04.701 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T06:02:04.701 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T06:02:04.701 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T06:02:04.754 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T06:02:04.771 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T06:02:04.771 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T06:02:04.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T06:02:04.798 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:02:04.798 INFO:tasks.ceph:role: osd.0 2024-04-16T06:02:04.799 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T06:02:04.799 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:02:04.874 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:02:05.401 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-16T06:02:05.403 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-16T06:02:05.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T06:02:05.564 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T06:02:05.601 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T06:02:05.671 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:02:05.671 INFO:tasks.ceph:role: osd.1 2024-04-16T06:02:05.671 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T06:02:05.671 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T06:02:05.747 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:02:05.747 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:02:05.747 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:02:05.748 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:02:06.202 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-16T06:02:06.204 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-16T06:02:06.204 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T06:02:06.299 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T06:02:06.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T06:02:06.404 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:02:06.404 INFO:tasks.ceph:role: osd.2 2024-04-16T06:02:06.404 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T06:02:06.404 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T06:02:06.480 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:02:06.480 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:02:06.480 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:02:06.481 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:02:06.961 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-04-16T06:02:06.963 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-04-16T06:02:06.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-16T06:02:07.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T06:02:07.138 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:07.478 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:07.479 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:07,482 seastar - Reactor backend: linux-aio 2024-04-16T06:02:07.493 INFO:teuthology.orchestra.run.smithi203.stderr:==29086==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:07.611 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:07,614 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:02:07.611 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:07,614 [shard 0:main] osd - get_early_config: set --cpuset 0-1 2024-04-16T06:02:07.612 INFO:teuthology.orchestra.run.smithi203.stderr:DEBUG 2024-04-16 06:02:07,615 [shard 0:main] seastar - reactor::drain 2024-04-16T06:02:08.155 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:08.156 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:08.156 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:08.156 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:08.156 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:08.156 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:08.222 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:08.226 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:08,229 seastar - Reactor backend: linux-aio 2024-04-16T06:02:08.233 INFO:teuthology.orchestra.run.smithi203.stderr:==29084==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:11.699 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:11.700 INFO:teuthology.orchestra.run.smithi203.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:11.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T06:02:11.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:12.051 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:12.052 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:12,055 seastar - Reactor backend: linux-aio 2024-04-16T06:02:12.057 INFO:teuthology.orchestra.run.smithi203.stderr:==29892==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:12.175 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:12,178 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:02:12.175 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:12,178 [shard 0:main] osd - get_early_config: set --cpuset 2-3 2024-04-16T06:02:12.175 INFO:teuthology.orchestra.run.smithi203.stderr:DEBUG 2024-04-16 06:02:12,179 [shard 0:main] seastar - reactor::drain 2024-04-16T06:02:12.723 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:12.723 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:12.723 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:12.724 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:12.724 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:12.724 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:12.777 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:12.781 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:12,784 seastar - Reactor backend: linux-aio 2024-04-16T06:02:12.788 INFO:teuthology.orchestra.run.smithi203.stderr:==29890==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:16.251 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:16.252 INFO:teuthology.orchestra.run.smithi203.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:16.276 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T06:02:16.304 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:16.605 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:16.606 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:16,609 seastar - Reactor backend: linux-aio 2024-04-16T06:02:16.612 INFO:teuthology.orchestra.run.smithi203.stderr:==30698==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:16.718 INFO:teuthology.orchestra.run.smithi203.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a202b 0xb874970 0xb875eec 0xb876102 0xb876258 0xb876721 0x54daf 0x2eaf 0x4a09d 0x35b7c 0x3629e 0x34877b9 0x5dac77c 0x5dacbd0 0x5dad18b 0x5dad2b1 0x5e09234 0x5d87db0 0x5d882e4 0x5bccfdf 0x5be80a3 0x6ec2944 0x6ed4117 0x6ed55b4 0x6ed5639 0x6ed573b 0x6ed5833 0x6ed5a40 0x6ed5c26 0x6ed5f75 0x6ed6319 0x6ed6549 0x6ed67b3 0x6ed6a80 0x6ed6c9e 0x6ed6d56 0x6ed6e1c 0x6ed6efd 0x5a358de 0x5a359e8 0x5a35aa4 0x5a35b60 0x5a35c1c 0x5a35d69 0x5a35ef4 0x5a362b5 0x5a521f6 0xb8876e3 0xb8a197d 0xb942a0f 0xb94405d 0xb6206c1 0xb62103d 0x37d7bfc 0x37d8755 0x3678150 0x3feaf 0x3ff5f 0x3462444 2024-04-16T06:02:16.718 INFO:teuthology.orchestra.run.smithi203.stderr:kernel callstack: 2024-04-16T06:02:16.731 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:16,733 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:02:16.731 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:16,734 [shard 0:main] osd - get_early_config: set --cpuset 4-5 2024-04-16T06:02:16.732 INFO:teuthology.orchestra.run.smithi203.stderr:DEBUG 2024-04-16 06:02:16,735 [shard 0:main] seastar - reactor::drain 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:17.269 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:17.327 INFO:teuthology.orchestra.run.smithi203.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:17.331 INFO:teuthology.orchestra.run.smithi203.stderr:INFO 2024-04-16 06:02:17,334 seastar - Reactor backend: linux-aio 2024-04-16T06:02:17.338 INFO:teuthology.orchestra.run.smithi203.stderr:==30696==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:20.787 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:20.787 INFO:teuthology.orchestra.run.smithi203.stderr:Suppressions used: 2024-04-16T06:02:20.788 INFO:teuthology.orchestra.run.smithi203.stderr: count bytes template 2024-04-16T06:02:20.788 INFO:teuthology.orchestra.run.smithi203.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:20.788 INFO:teuthology.orchestra.run.smithi203.stderr:----------------------------------------------------- 2024-04-16T06:02:20.788 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T06:02:20.788 INFO:teuthology.orchestra.run.smithi203.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 8af020fd-9691-42e7-ac58-13efaaf4e615 2024-04-16T06:02:20.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T06:02:20.840 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T06:02:20.840 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:20.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T06:02:20.908 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:20.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T06:02:20.977 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:20.977 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T06:02:21.044 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:21.045 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T06:02:21.112 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:21.112 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T06:02:21.169 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T06:02:21.169 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBcFB5mP3HTKRAA7GPLeQWB1i701yCCum0f8A== 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBgFB5mjKSdFhAAqXw8O+nYfZGC4APNiCKsJg== 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBkFB5m+vjdNxAA1xRHAfZWA1WrBMzhmOXeCA== 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBpFB5mAWDXHBAAYIs7K4rrgNydeP7pbo1prw== 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-04-16T06:02:21.235 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBcFB5mfv8lLRAAPTQKKkuHWdSbZYRmITHJrQ== 2024-04-16T06:02:21.236 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T06:02:21.340 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:21.403 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:21.466 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:21.529 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T06:02:21.595 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T06:02:21.595 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T06:02:21.623 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T06:02:21.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:21.731+0000 7f1b9c45ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.729 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:21.731+0000 7f1b9c45ca80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.743 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T06:02:21.771 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:21.826 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T06:02:21.826 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T06:02:21.826 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T06:02:21.869 INFO:tasks.ceph.mon.a:Started 2024-04-16T06:02:21.869 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T06:02:21.869 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T06:02:21.869 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T06:02:21.871 INFO:tasks.ceph.mgr.x:Started 2024-04-16T06:02:21.871 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T06:02:21.871 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T06:02:21.956 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:02:21.958+0000 7f7f6a25ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.956 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:02:21.958+0000 7f7f6a25ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.956 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:21.958+0000 7f443cf69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.956 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:02:21.958+0000 7f7f6a25ba80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:21.983+0000 7f6561b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:21.993 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:21.995+0000 7f6561b1d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:22.063 INFO:teuthology.orchestra.run.smithi203.stderr:adjusted tunables profile to default 2024-04-16T06:02:22.074 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T06:02:22.074 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T06:02:22.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T06:02:22.160 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:22.163+0000 7f443cf69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:22.160 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:22.163+0000 7f443cf69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:22.185 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:22.182+0000 7fd10ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:22.189 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:22.191+0000 7fd10ba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:02:22.275+0000 7f7f60ff4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr:feature with important data. 2024-04-16T06:02:22.273 INFO:tasks.ceph.mon.a.smithi203.stderr: 2024-04-16T06:02:22.642 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:22.645+0000 7f443cf69000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T06:02:22.696 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:22.699+0000 7f443cf69000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T06:02:23.075 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T06:02:23.076 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:23.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T06:02:23.104 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:23.104 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T06:02:23.173 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:02:23.173 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T06:02:23.231 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.233+0000 7f443cf69000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T06:02:23.242 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 33281e3c-7fe2-43fa-9a6e-57425b3bfca0 0 2024-04-16T06:02:23.287 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.290+0000 7f443cf69000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T06:02:23.335 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.337+0000 7f443cf69000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T06:02:23.393 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.396+0000 7f443cf69000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T06:02:23.394 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.391+0000 7f0fc02ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.398 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.400+0000 7f0fc02ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.439 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.441+0000 7f443cf69000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T06:02:23.479 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-04-16T06:02:23.490 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 10e714d9-dcf3-421b-b25b-309914533c99 1 2024-04-16T06:02:23.562 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.564+0000 7f443cf69000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T06:02:23.637 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.639+0000 7f443cf69000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T06:02:23.643 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.639+0000 7f4d230ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.649+0000 7f4d230ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.686 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.689+0000 7f443cf69000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T06:02:23.728 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-04-16T06:02:23.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 55ce6e8a-830d-4f7a-917b-5d33cc08e65a 2 2024-04-16T06:02:23.742 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.744+0000 7f443cf69000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T06:02:23.792 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:23.794+0000 7f443cf69000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T06:02:23.854 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.850+0000 7f0b06d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:23.859+0000 7f0b06d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:23.937 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-04-16T06:02:23.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T06:02:23.981 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T06:02:23.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T06:02:24.024 INFO:tasks.ceph.osd.0:Started 2024-04-16T06:02:24.024 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T06:02:24.056 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T06:02:24.056 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T06:02:24.099 INFO:tasks.ceph.osd.1:Started 2024-04-16T06:02:24.099 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T06:02:24.132 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T06:02:24.132 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T06:02:24.176 INFO:tasks.ceph.osd.2:Started 2024-04-16T06:02:24.176 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:24.212 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.214+0000 7f443cf69000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T06:02:24.360 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:24.357+0000 7fec89d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:24.365 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:24.367+0000 7fec89d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:24.440 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.442+0000 7f443cf69000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T06:02:24.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:24.467 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":6,"fsid":"8af020fd-9691-42e7-ac58-13efaaf4e615","created":"2024-04-16T06:02:21.981254+0000","modified":"2024-04-16T06:02:23.939795+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T06:02:23.939795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"33281e3c-7fe2-43fa-9a6e-57425b3bfca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"10e714d9-dcf3-421b-b25b-309914533c99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"55ce6e8a-830d-4f7a-917b-5d33cc08e65a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:24.483 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T06:02:24.483 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T06:02:24.567 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.569+0000 7f443cf69000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T06:02:24.616 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.618+0000 7f443cf69000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T06:02:24.738 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.740+0000 7f443cf69000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T06:02:24.785 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:24.844 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:24.847+0000 7f443cf69000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T06:02:24.892 INFO:teuthology.misc.health.smithi203.stderr:2024-04-16T06:02:24.890+0000 7fd988f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:24.896 INFO:teuthology.misc.health.smithi203.stderr:2024-04-16T06:02:24.899+0000 7fd988f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:24.968 INFO:teuthology.misc.health.smithi203.stdout: 2024-04-16T06:02:24.969 INFO:teuthology.misc.health.smithi203.stdout:{"epoch":6,"fsid":"8af020fd-9691-42e7-ac58-13efaaf4e615","created":"2024-04-16T06:02:21.981254+0000","modified":"2024-04-16T06:02:23.939795+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T06:02:23.939795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"33281e3c-7fe2-43fa-9a6e-57425b3bfca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"10e714d9-dcf3-421b-b25b-309914533c99","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"55ce6e8a-830d-4f7a-917b-5d33cc08e65a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:24.980 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-16T06:02:25.022 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:25.024+0000 7f443cf69000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T06:02:25.069 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:25.072+0000 7f443cf69000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T06:02:25.241 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:25.243+0000 7f443cf69000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T06:02:26.109 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:26.111+0000 7f443cf69000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T06:02:26.397 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:26.399+0000 7f443cf69000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T06:02:26.453 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:26.455+0000 7f443cf69000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T06:02:26.661 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:26.664+0000 7f443cf69000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T06:02:27.671 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:02:27.673+0000 7f43dc0f4640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T06:02:31.282 DEBUG:teuthology.orchestra.run.smithi203:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:31.394 INFO:teuthology.misc.health.smithi203.stderr:2024-04-16T06:02:31.393+0000 7fd4f515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:31.398 INFO:teuthology.misc.health.smithi203.stderr:2024-04-16T06:02:31.400+0000 7fd4f515d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:31.586 INFO:teuthology.misc.health.smithi203.stdout: 2024-04-16T06:02:31.586 INFO:teuthology.misc.health.smithi203.stdout:{"epoch":9,"fsid":"8af020fd-9691-42e7-ac58-13efaaf4e615","created":"2024-04-16T06:02:21.981254+0000","modified":"2024-04-16T06:02:30.738881+0000","last_up_change":"2024-04-16T06:02:28.731427+0000","last_in_change":"2024-04-16T06:02:23.939795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:30.690730+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33281e3c-7fe2-43fa-9a6e-57425b3bfca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":2317714937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2317714937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":2317714937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2317714937}]},"public_addr":"172.21.15.203:6803/2317714937","cluster_addr":"172.21.15.203:6802/2317714937","heartbeat_back_addr":"172.21.15.203:6815/2317714937","heartbeat_front_addr":"172.21.15.203:6814/2317714937","state":["exists","up"]},{"osd":1,"uuid":"10e714d9-dcf3-421b-b25b-309914533c99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":2521931534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2521931534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":2521931534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2521931534}]},"public_addr":"172.21.15.203:6805/2521931534","cluster_addr":"172.21.15.203:6806/2521931534","heartbeat_back_addr":"172.21.15.203:6811/2521931534","heartbeat_front_addr":"172.21.15.203:6808/2521931534","state":["exists","up"]},{"osd":2,"uuid":"55ce6e8a-830d-4f7a-917b-5d33cc08e65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6807","nonce":241280154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":241280154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":241280154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":241280154}]},"public_addr":"172.21.15.203:6807/241280154","cluster_addr":"172.21.15.203:6804/241280154","heartbeat_back_addr":"172.21.15.203:6809/241280154","heartbeat_front_addr":"172.21.15.203:6810/241280154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:31.598 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-16T06:02:31.598 INFO:tasks.ceph:Creating RBD pool 2024-04-16T06:02:31.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T06:02:31.712 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:31.708+0000 7fd53d31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:31.716 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:31.718+0000 7fd53d31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:32.791 INFO:teuthology.orchestra.run.smithi203.stderr:pool 'rbd' created 2024-04-16T06:02:32.809 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T06:02:32.955 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:32.945+0000 7f8ca6b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:32.959 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:32.961+0000 7f8ca6b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:33.793 INFO:teuthology.orchestra.run.smithi203.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T06:02:33.814 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T06:02:33.814 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T06:02:33.814 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T06:02:33.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T06:02:33.815 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T06:02:33.963 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:33.961+0000 7f16b946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:33.966 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:33.968+0000 7f16b946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.203 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:34.215 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":3960487642},{"type":"v1","addr":"172.21.15.203:6801","nonce":3960487642}]},"active_addr":"172.21.15.203:6801/3960487642","active_change":"2024-04-16T06:02:26.666906+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":654749735}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":2214801428}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3328860226}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":900814297}]}]} 2024-04-16T06:02:34.218 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T06:02:34.218 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T06:02:34.219 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:34.331 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.330+0000 7f737f864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.337+0000 7f737f864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.522 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:34.522 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":12,"fsid":"8af020fd-9691-42e7-ac58-13efaaf4e615","created":"2024-04-16T06:02:21.981254+0000","modified":"2024-04-16T06:02:33.794992+0000","last_up_change":"2024-04-16T06:02:28.731427+0000","last_in_change":"2024-04-16T06:02:23.939795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:30.690730+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T06:02:32.059343+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33281e3c-7fe2-43fa-9a6e-57425b3bfca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":2317714937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2317714937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":2317714937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2317714937}]},"public_addr":"172.21.15.203:6803/2317714937","cluster_addr":"172.21.15.203:6802/2317714937","heartbeat_back_addr":"172.21.15.203:6815/2317714937","heartbeat_front_addr":"172.21.15.203:6814/2317714937","state":["exists","up"]},{"osd":1,"uuid":"10e714d9-dcf3-421b-b25b-309914533c99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":2521931534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2521931534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":2521931534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2521931534}]},"public_addr":"172.21.15.203:6805/2521931534","cluster_addr":"172.21.15.203:6806/2521931534","heartbeat_back_addr":"172.21.15.203:6811/2521931534","heartbeat_front_addr":"172.21.15.203:6808/2521931534","state":["exists","up"]},{"osd":2,"uuid":"55ce6e8a-830d-4f7a-917b-5d33cc08e65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6807","nonce":241280154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":241280154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":241280154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":241280154}]},"public_addr":"172.21.15.203:6807/241280154","cluster_addr":"172.21.15.203:6804/241280154","heartbeat_back_addr":"172.21.15.203:6809/241280154","heartbeat_front_addr":"172.21.15.203:6810/241280154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:34.535 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T06:02:34.536 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:34.647 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.646+0000 7fe6f7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.651 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.653+0000 7fe6f7c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.853 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:34.853 INFO:teuthology.orchestra.run.smithi203.stdout:{"epoch":13,"fsid":"8af020fd-9691-42e7-ac58-13efaaf4e615","created":"2024-04-16T06:02:21.981254+0000","modified":"2024-04-16T06:02:34.796575+0000","last_up_change":"2024-04-16T06:02:28.731427+0000","last_in_change":"2024-04-16T06:02:23.939795+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:30.690730+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T06:02:32.059343+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"33281e3c-7fe2-43fa-9a6e-57425b3bfca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6803","nonce":2317714937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2317714937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":2317714937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":2317714937}]},"public_addr":"172.21.15.203:6803/2317714937","cluster_addr":"172.21.15.203:6802/2317714937","heartbeat_back_addr":"172.21.15.203:6815/2317714937","heartbeat_front_addr":"172.21.15.203:6814/2317714937","state":["exists","up"]},{"osd":1,"uuid":"10e714d9-dcf3-421b-b25b-309914533c99","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6805","nonce":2521931534}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2521931534}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6811","nonce":2521931534}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":2521931534}]},"public_addr":"172.21.15.203:6805/2521931534","cluster_addr":"172.21.15.203:6806/2521931534","heartbeat_back_addr":"172.21.15.203:6811/2521931534","heartbeat_front_addr":"172.21.15.203:6808/2521931534","state":["exists","up"]},{"osd":2,"uuid":"55ce6e8a-830d-4f7a-917b-5d33cc08e65a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6807","nonce":241280154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":241280154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":241280154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6810","nonce":241280154}]},"public_addr":"172.21.15.203:6807/241280154","cluster_addr":"172.21.15.203:6804/241280154","heartbeat_back_addr":"172.21.15.203:6809/241280154","heartbeat_front_addr":"172.21.15.203:6810/241280154","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:34.866 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T06:02:34.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T06:02:34.867 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T06:02:34.986 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.983+0000 7fa6aba06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.992+0000 7fa6aba06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.990 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.983+0000 7f023115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.991 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.983+0000 7f339d6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.994 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.996+0000 7f023115d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:34.995 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:34.998+0000 7f339d6ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.024 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.025 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T06:02:35.029 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.029 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T06:02:35.037 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T06:02:35.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.138+0000 7fb6b1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.146+0000 7fb6b1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.196 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.193+0000 7f857cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.195+0000 7fed9da2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.200 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.202+0000 7f857cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.201 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.204+0000 7fed9da2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.350 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2024-04-16T06:02:35.365 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:35.397 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.403 INFO:teuthology.orchestra.run.smithi203.stdout:34359738374 2024-04-16T06:02:35.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2024-04-16T06:02:35.411 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:35.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2024-04-16T06:02:35.416 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:35.417 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T06:02:35.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T06:02:35.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.568+0000 7f17fa0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.573 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.575+0000 7f17fa0ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.755 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:35.756 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2024-04-16T06:02:35.768 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T06:02:35.027943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.824093+0000","last_change":"2024-04-16T06:02:33.890817+0000","last_active":"2024-04-16T06:02:34.824093+0000","last_peered":"2024-04-16T06:02:34.824093+0000","last_clean":"2024-04-16T06:02:34.824093+0000","last_became_active":"2024-04-16T06:02:33.885041+0000","last_became_peered":"2024-04-16T06:02:33.885041+0000","last_unstale":"2024-04-16T06:02:34.824093+0000","last_undegraded":"2024-04-16T06:02:34.824093+0000","last_fullsized":"2024-04-16T06:02:34.824093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.830736+0000","last_change":"2024-04-16T06:02:33.911463+0000","last_active":"2024-04-16T06:02:34.830736+0000","last_peered":"2024-04-16T06:02:34.830736+0000","last_clean":"2024-04-16T06:02:34.830736+0000","last_became_active":"2024-04-16T06:02:33.886621+0000","last_became_peered":"2024-04-16T06:02:33.886621+0000","last_unstale":"2024-04-16T06:02:34.830736+0000","last_undegraded":"2024-04-16T06:02:34.830736+0000","last_fullsized":"2024-04-16T06:02:34.830736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.820998+0000","last_change":"2024-04-16T06:02:33.875946+0000","last_active":"2024-04-16T06:02:34.820998+0000","last_peered":"2024-04-16T06:02:34.820998+0000","last_clean":"2024-04-16T06:02:34.820998+0000","last_became_active":"2024-04-16T06:02:33.869606+0000","last_became_peered":"2024-04-16T06:02:33.869606+0000","last_unstale":"2024-04-16T06:02:34.820998+0000","last_undegraded":"2024-04-16T06:02:34.820998+0000","last_fullsized":"2024-04-16T06:02:34.820998+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.832070+0000","last_change":"2024-04-16T06:02:33.892373+0000","last_active":"2024-04-16T06:02:34.832070+0000","last_peered":"2024-04-16T06:02:34.832070+0000","last_clean":"2024-04-16T06:02:34.832070+0000","last_became_active":"2024-04-16T06:02:33.881167+0000","last_became_peered":"2024-04-16T06:02:33.881167+0000","last_unstale":"2024-04-16T06:02:34.832070+0000","last_undegraded":"2024-04-16T06:02:34.832070+0000","last_fullsized":"2024-04-16T06:02:34.832070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.900645+0000","last_change":"2024-04-16T06:02:33.901590+0000","last_active":"2024-04-16T06:02:33.900645+0000","last_peered":"2024-04-16T06:02:33.900645+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.896937+0000","last_became_peered":"2024-04-16T06:02:33.896937+0000","last_unstale":"2024-04-16T06:02:33.900645+0000","last_undegraded":"2024-04-16T06:02:33.900645+0000","last_fullsized":"2024-04-16T06:02:33.900645+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.894131+0000","last_change":"2024-04-16T06:02:33.895717+0000","last_active":"2024-04-16T06:02:33.894131+0000","last_peered":"2024-04-16T06:02:33.894131+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.889888+0000","last_became_peered":"2024-04-16T06:02:33.889888+0000","last_unstale":"2024-04-16T06:02:33.894131+0000","last_undegraded":"2024-04-16T06:02:33.894131+0000","last_fullsized":"2024-04-16T06:02:33.894131+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.889032+0000","last_change":"2024-04-16T06:02:33.890560+0000","last_active":"2024-04-16T06:02:33.889032+0000","last_peered":"2024-04-16T06:02:33.889032+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.884913+0000","last_became_peered":"2024-04-16T06:02:33.884913+0000","last_unstale":"2024-04-16T06:02:33.889032+0000","last_undegraded":"2024-04-16T06:02:33.889032+0000","last_fullsized":"2024-04-16T06:02:33.889032+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.819981+0000","last_change":"2024-04-16T06:02:33.858523+0000","last_active":"2024-04-16T06:02:34.819981+0000","last_peered":"2024-04-16T06:02:34.819981+0000","last_clean":"2024-04-16T06:02:34.819981+0000","last_became_active":"2024-04-16T06:02:33.852837+0000","last_became_peered":"2024-04-16T06:02:33.852837+0000","last_unstale":"2024-04-16T06:02:34.819981+0000","last_undegraded":"2024-04-16T06:02:34.819981+0000","last_fullsized":"2024-04-16T06:02:34.819981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.825599+0000","last_change":"2024-04-16T06:02:31.783712+0000","last_active":"2024-04-16T06:02:34.825599+0000","last_peered":"2024-04-16T06:02:34.825599+0000","last_clean":"2024-04-16T06:02:34.825599+0000","last_became_active":"2024-04-16T06:02:31.776848+0000","last_became_peered":"2024-04-16T06:02:31.776848+0000","last_unstale":"2024-04-16T06:02:34.825599+0000","last_undegraded":"2024-04-16T06:02:34.825599+0000","last_fullsized":"2024-04-16T06:02:34.825599+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:30.738881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:30.738881+0000","last_clean_scrub_stamp":"2024-04-16T06:02:30.738881+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T06:02:35.769 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T06:02:35.881 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.879+0000 7f4ad4b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:35.884 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:35.886+0000 7f4ad4b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:36.067 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:36.067 INFO:teuthology.orchestra.run.smithi203.stderr:dumped all 2024-04-16T06:02:36.079 INFO:teuthology.orchestra.run.smithi203.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T06:02:35.027943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82692,"kb_used_data":1460,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490108,"statfs":{"total":322122547200,"available":322037870592,"internally_reserved":0,"allocated":1495040,"data_stored":1251956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.824093+0000","last_change":"2024-04-16T06:02:33.890817+0000","last_active":"2024-04-16T06:02:34.824093+0000","last_peered":"2024-04-16T06:02:34.824093+0000","last_clean":"2024-04-16T06:02:34.824093+0000","last_became_active":"2024-04-16T06:02:33.885041+0000","last_became_peered":"2024-04-16T06:02:33.885041+0000","last_unstale":"2024-04-16T06:02:34.824093+0000","last_undegraded":"2024-04-16T06:02:34.824093+0000","last_fullsized":"2024-04-16T06:02:34.824093+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.830736+0000","last_change":"2024-04-16T06:02:33.911463+0000","last_active":"2024-04-16T06:02:34.830736+0000","last_peered":"2024-04-16T06:02:34.830736+0000","last_clean":"2024-04-16T06:02:34.830736+0000","last_became_active":"2024-04-16T06:02:33.886621+0000","last_became_peered":"2024-04-16T06:02:33.886621+0000","last_unstale":"2024-04-16T06:02:34.830736+0000","last_undegraded":"2024-04-16T06:02:34.830736+0000","last_fullsized":"2024-04-16T06:02:34.830736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.820998+0000","last_change":"2024-04-16T06:02:33.875946+0000","last_active":"2024-04-16T06:02:34.820998+0000","last_peered":"2024-04-16T06:02:34.820998+0000","last_clean":"2024-04-16T06:02:34.820998+0000","last_became_active":"2024-04-16T06:02:33.869606+0000","last_became_peered":"2024-04-16T06:02:33.869606+0000","last_unstale":"2024-04-16T06:02:34.820998+0000","last_undegraded":"2024-04-16T06:02:34.820998+0000","last_fullsized":"2024-04-16T06:02:34.820998+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.832070+0000","last_change":"2024-04-16T06:02:33.892373+0000","last_active":"2024-04-16T06:02:34.832070+0000","last_peered":"2024-04-16T06:02:34.832070+0000","last_clean":"2024-04-16T06:02:34.832070+0000","last_became_active":"2024-04-16T06:02:33.881167+0000","last_became_peered":"2024-04-16T06:02:33.881167+0000","last_unstale":"2024-04-16T06:02:34.832070+0000","last_undegraded":"2024-04-16T06:02:34.832070+0000","last_fullsized":"2024-04-16T06:02:34.832070+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.900645+0000","last_change":"2024-04-16T06:02:33.901590+0000","last_active":"2024-04-16T06:02:33.900645+0000","last_peered":"2024-04-16T06:02:33.900645+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.896937+0000","last_became_peered":"2024-04-16T06:02:33.896937+0000","last_unstale":"2024-04-16T06:02:33.900645+0000","last_undegraded":"2024-04-16T06:02:33.900645+0000","last_fullsized":"2024-04-16T06:02:33.900645+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.894131+0000","last_change":"2024-04-16T06:02:33.895717+0000","last_active":"2024-04-16T06:02:33.894131+0000","last_peered":"2024-04-16T06:02:33.894131+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.889888+0000","last_became_peered":"2024-04-16T06:02:33.889888+0000","last_unstale":"2024-04-16T06:02:33.894131+0000","last_undegraded":"2024-04-16T06:02:33.894131+0000","last_fullsized":"2024-04-16T06:02:33.894131+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:33.889032+0000","last_change":"2024-04-16T06:02:33.890560+0000","last_active":"2024-04-16T06:02:33.889032+0000","last_peered":"2024-04-16T06:02:33.889032+0000","last_clean":"2024-04-16T06:02:32.791002+0000","last_became_active":"2024-04-16T06:02:33.884913+0000","last_became_peered":"2024-04-16T06:02:33.884913+0000","last_unstale":"2024-04-16T06:02:33.889032+0000","last_undegraded":"2024-04-16T06:02:33.889032+0000","last_fullsized":"2024-04-16T06:02:33.889032+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.819981+0000","last_change":"2024-04-16T06:02:33.858523+0000","last_active":"2024-04-16T06:02:34.819981+0000","last_peered":"2024-04-16T06:02:34.819981+0000","last_clean":"2024-04-16T06:02:34.819981+0000","last_became_active":"2024-04-16T06:02:33.852837+0000","last_became_peered":"2024-04-16T06:02:33.852837+0000","last_unstale":"2024-04-16T06:02:34.819981+0000","last_undegraded":"2024-04-16T06:02:34.819981+0000","last_fullsized":"2024-04-16T06:02:34.819981+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:32.791002+0000","last_clean_scrub_stamp":"2024-04-16T06:02:32.791002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-16T06:02:34.825599+0000","last_change":"2024-04-16T06:02:31.783712+0000","last_active":"2024-04-16T06:02:34.825599+0000","last_peered":"2024-04-16T06:02:34.825599+0000","last_clean":"2024-04-16T06:02:34.825599+0000","last_became_active":"2024-04-16T06:02:31.776848+0000","last_became_peered":"2024-04-16T06:02:31.776848+0000","last_unstale":"2024-04-16T06:02:34.825599+0000","last_undegraded":"2024-04-16T06:02:34.825599+0000","last_fullsized":"2024-04-16T06:02:34.825599+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:30.738881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:30.738881+0000","last_clean_scrub_stamp":"2024-04-16T06:02:30.738881+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27052,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830548,"statfs":{"total":107374182400,"available":107346481152,"internally_reserved":0,"allocated":102400,"data_stored":23740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27820,"kb_used_data":680,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829780,"statfs":{"total":107374182400,"available":107345694720,"internally_reserved":0,"allocated":696320,"data_stored":614108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T06:02:36.080 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T06:02:36.080 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T06:02:36.080 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T06:02:36.080 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T06:02:36.190 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:36.190+0000 7f9d3f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:36.194 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T06:02:36.196+0000 7f9d3f26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:36.418 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-04-16T06:02:36.419 INFO:teuthology.orchestra.run.smithi203.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T06:02:36.431 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T06:02:36.431 INFO:teuthology.run_tasks:Running task workunit... 2024-04-16T06:02:36.443 INFO:tasks.workunit:Pulling workunits from ref c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:02:36.443 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-16T06:02:36.444 DEBUG:teuthology.orchestra.run.smithi203:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:36.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:02:36.459 INFO:teuthology.orchestra.run.smithi203.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-16T06:02:36.459 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:36.515 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:36.515 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-16T06:02:36.571 INFO:tasks.workunit:timeout=3h 2024-04-16T06:02:36.571 INFO:tasks.workunit:cleanup=True 2024-04-16T06:02:36.572 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:02:36.628 INFO:tasks.workunit.client.0.smithi203.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr:Note: switching to 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'. 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr:state without impacting any branches by switching back to a branch. 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.453 INFO:tasks.workunit.client.0.smithi203.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: git switch -c 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr:Or undo this operation with: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: git switch - 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr: 2024-04-16T06:03:59.454 INFO:tasks.workunit.client.0.smithi203.stderr:HEAD is now at c0a287816af seastar: submodule update 2024-04-16T06:03:59.459 DEBUG:teuthology.orchestra.run.smithi203:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-16T06:03:59.523 INFO:tasks.workunit.client.0.smithi203.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-16T06:03:59.526 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T06:03:59.526 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-16T06:04:00.181 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-16T06:04:00.252 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-16T06:04:00.293 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T06:04:00.294 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T06:04:00.295 INFO:tasks.workunit.client.0.smithi203.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-16T06:04:00.336 INFO:tasks.workunit.client.0.smithi203.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T06:04:00.339 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-04-16T06:04:00.339 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-16T06:04:00.395 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-04-16T06:04:00.396 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-04-16T06:04:00.397 DEBUG:teuthology.orchestra.run.smithi203:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-04-16T06:04:00.472 INFO:tasks.workunit.client.0.smithi203.stdout:Running main() from gmock_main.cc 2024-04-16T06:04:00.472 INFO:tasks.workunit.client.0.smithi203.stdout:[==========] Running 70 tests from 1 test suite. 2024-04-16T06:04:00.473 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] Global test environment set-up. 2024-04-16T06:04:00.473 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] 70 tests from TestClsRbd 2024-04-16T06:04:02.019 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_all_features 2024-04-16T06:04:07.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T06:04:13.179 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-16T06:04:18.582 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-16T06:04:23.985 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-16T06:04:29.227 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_all_features (27208 ms) 2024-04-16T06:04:29.228 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.copyup 2024-04-16T06:04:29.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-04-16T06:04:29.461 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.copyup (234 ms) 2024-04-16T06:04:29.461 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-04-16T06:04:29.564 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.sparse_copyup (103 ms) 2024-04-16T06:04:29.564 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-04-16T06:04:29.625 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_and_set_id (61 ms) 2024-04-16T06:04:29.625 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-04-16T06:04:29.719 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.add_remove_child (94 ms) 2024-04-16T06:04:29.719 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.directory_methods 2024-04-16T06:04:29.996 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.directory_methods (277 ms) 2024-04-16T06:04:29.996 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.create 2024-04-16T06:04:30.167 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.create (171 ms) 2024-04-16T06:04:30.167 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_features 2024-04-16T06:04:30.184 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_features (17 ms) 2024-04-16T06:04:30.184 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-04-16T06:04:30.203 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_object_prefix (19 ms) 2024-04-16T06:04:30.203 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-04-16T06:04:30.217 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_create_timestamp (14 ms) 2024-04-16T06:04:30.217 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-04-16T06:04:30.230 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_access_timestamp (13 ms) 2024-04-16T06:04:30.230 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-04-16T06:04:30.243 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_modify_timestamp (13 ms) 2024-04-16T06:04:30.243 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-04-16T06:04:30.273 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_data_pool (30 ms) 2024-04-16T06:04:30.273 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.get_size 2024-04-16T06:04:30.317 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.get_size (45 ms) 2024-04-16T06:04:30.317 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.set_size 2024-04-16T06:04:30.366 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.set_size (49 ms) 2024-04-16T06:04:30.366 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.protection_status 2024-04-16T06:04:30.537 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.protection_status (170 ms) 2024-04-16T06:04:30.538 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-04-16T06:04:30.610 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.snapshot_limits (72 ms) 2024-04-16T06:04:30.610 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.parents_v1 2024-04-16T06:04:31.035 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.parents_v1 (425 ms) 2024-04-16T06:04:31.035 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.parents_v2 2024-04-16T06:04:31.340 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.parents_v2 (305 ms) 2024-04-16T06:04:31.340 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.snapshots 2024-04-16T06:04:31.625 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.snapshots (286 ms) 2024-04-16T06:04:31.626 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.snapid_race 2024-04-16T06:04:31.655 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.snapid_race (29 ms) 2024-04-16T06:04:31.655 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.stripingv2 2024-04-16T06:04:31.731 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.stripingv2 (77 ms) 2024-04-16T06:04:31.732 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_save 2024-04-16T06:04:31.745 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_save (14 ms) 2024-04-16T06:04:31.745 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-04-16T06:04:31.790 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_resize (44 ms) 2024-04-16T06:04:31.790 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_update 2024-04-16T06:04:31.834 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_update (44 ms) 2024-04-16T06:04:31.835 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-04-16T06:04:31.838 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_load_enoent (4 ms) 2024-04-16T06:04:31.838 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-04-16T06:04:31.873 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_snap_add (34 ms) 2024-04-16T06:04:31.873 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-04-16T06:04:31.909 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.object_map_snap_remove (36 ms) 2024-04-16T06:04:31.909 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.flags 2024-04-16T06:04:31.977 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.flags (68 ms) 2024-04-16T06:04:31.977 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.metadata 2024-04-16T06:04:32.275 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.metadata (297 ms) 2024-04-16T06:04:32.275 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.set_features 2024-04-16T06:04:32.344 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.set_features (70 ms) 2024-04-16T06:04:32.344 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror 2024-04-16T06:04:32.648 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror (304 ms) 2024-04-16T06:04:32.648 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror_image 2024-04-16T06:04:32.796 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror_image (148 ms) 2024-04-16T06:04:32.796 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-04-16T06:04:34.790 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-04-16T06:04:40.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-04-16T06:04:45.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2024-04-16T06:04:50.999 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2024-04-16T06:04:56.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-04-16T06:05:01.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2024-04-16T06:05:07.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2024-04-16T06:05:11.234 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror_image_status (38437 ms) 2024-04-16T06:05:11.234 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-04-16T06:05:12.362 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror_image_map (1129 ms) 2024-04-16T06:05:12.362 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-04-16T06:05:12.444 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror_instances (82 ms) 2024-04-16T06:05:12.444 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-04-16T06:05:12.550 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.mirror_snapshot (106 ms) 2024-04-16T06:05:12.550 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-04-16T06:05:12.575 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_dir_list (26 ms) 2024-04-16T06:05:12.576 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-04-16T06:05:12.595 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_dir_add (19 ms) 2024-04-16T06:05:12.596 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-04-16T06:05:12.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-04-16T06:05:12.622 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.dir_add_already_existing (28 ms) 2024-04-16T06:05:12.622 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-04-16T06:05:12.679 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_dir_rename (56 ms) 2024-04-16T06:05:12.679 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-04-16T06:05:12.712 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_dir_remove (33 ms) 2024-04-16T06:05:12.712 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-04-16T06:05:12.751 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (39 ms) 2024-04-16T06:05:12.751 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_image_add 2024-04-16T06:05:12.770 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_image_add (19 ms) 2024-04-16T06:05:12.770 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-04-16T06:05:12.808 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_image_remove (38 ms) 2024-04-16T06:05:12.808 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_image_list 2024-04-16T06:05:12.836 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_image_list (28 ms) 2024-04-16T06:05:12.836 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-04-16T06:05:12.875 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_image_clean (39 ms) 2024-04-16T06:05:12.875 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.image_group_add 2024-04-16T06:05:12.903 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.image_group_add (28 ms) 2024-04-16T06:05:12.903 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-04-16T06:05:12.943 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.image_group_remove (40 ms) 2024-04-16T06:05:12.943 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.image_group_get 2024-04-16T06:05:12.971 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.image_group_get (29 ms) 2024-04-16T06:05:12.972 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-04-16T06:05:12.991 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (19 ms) 2024-04-16T06:05:12.991 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-04-16T06:05:13.005 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (14 ms) 2024-04-16T06:05:13.005 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-04-16T06:05:13.027 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (22 ms) 2024-04-16T06:05:13.027 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-04-16T06:05:13.048 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (22 ms) 2024-04-16T06:05:13.049 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-04-16T06:05:13.067 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_set (18 ms) 2024-04-16T06:05:13.067 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-04-16T06:05:13.098 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_list (29 ms) 2024-04-16T06:05:13.098 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-04-16T06:05:13.242 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (145 ms) 2024-04-16T06:05:13.242 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-04-16T06:05:15.464 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (2222 ms) 2024-04-16T06:05:15.465 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-04-16T06:05:15.496 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_remove (32 ms) 2024-04-16T06:05:15.496 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-04-16T06:05:15.516 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (20 ms) 2024-04-16T06:05:15.516 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.trash_methods 2024-04-16T06:05:15.609 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.trash_methods (93 ms) 2024-04-16T06:05:15.609 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.op_features 2024-04-16T06:05:15.694 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.op_features (85 ms) 2024-04-16T06:05:15.694 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.clone_parent 2024-04-16T06:05:15.892 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.clone_parent (198 ms) 2024-04-16T06:05:15.892 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-04-16T06:05:16.001 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.clone_parent_ns (109 ms) 2024-04-16T06:05:16.001 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.clone_child 2024-04-16T06:05:16.104 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.clone_child (103 ms) 2024-04-16T06:05:16.104 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-04-16T06:05:16.196 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.namespace_methods (92 ms) 2024-04-16T06:05:16.196 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.migration 2024-04-16T06:05:16.325 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.migration (129 ms) 2024-04-16T06:05:16.325 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.migration_v1 2024-04-16T06:05:16.403 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.migration_v1 (78 ms) 2024-04-16T06:05:16.403 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-04-16T06:05:18.014 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-04-16T06:05:18.201 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1798 ms) 2024-04-16T06:05:18.202 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] TestClsRbd.sparsify 2024-04-16T06:05:18.488 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] TestClsRbd.sparsify (287 ms) 2024-04-16T06:05:19.205 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] 70 tests from TestClsRbd (76469 ms total) 2024-04-16T06:05:19.205 INFO:tasks.workunit.client.0.smithi203.stdout: 2024-04-16T06:05:19.205 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] Global test environment tear-down 2024-04-16T06:05:19.205 INFO:tasks.workunit.client.0.smithi203.stdout:[==========] 70 tests from 1 test suite ran. (78733 ms total) 2024-04-16T06:05:19.205 INFO:tasks.workunit.client.0.smithi203.stdout:[ PASSED ] 70 tests. 2024-04-16T06:05:19.209 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-16T06:05:19.209 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-16T06:05:19.240 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2024-04-16T06:05:19.241 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2024-04-16T06:05:19.242 DEBUG:teuthology.orchestra.run.smithi203:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2024-04-16T06:05:19.322 INFO:tasks.workunit.client.0.smithi203.stdout:Running main() from gmock_main.cc 2024-04-16T06:05:19.322 INFO:tasks.workunit.client.0.smithi203.stdout:[==========] Running 11 tests from 1 test suite. 2024-04-16T06:05:19.322 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] Global test environment set-up. 2024-04-16T06:05:19.322 INFO:tasks.workunit.client.0.smithi203.stdout:[----------] 11 tests from ClsLock 2024-04-16T06:05:19.322 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] ClsLock.TestMultiLocking 2024-04-16T06:05:21.514 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:21.514 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: exclusive 2024-04-16T06:05:21.514 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:21.514 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:21.514 INFO:tasks.workunit.client.0.smithi203.stdout: client.4186 expiration=2024-04-16T06:07:21.487464+0000 addr=v1:172.21.15.203:0/2243330185 cookie= 2024-04-16T06:05:21.550 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:21.550 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: shared 2024-04-16T06:05:21.550 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:21.550 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:21.550 INFO:tasks.workunit.client.0.smithi203.stdout: client.4186 expiration=2024-04-16T06:07:21.540544+0000 addr=v1:172.21.15.203:0/2243330185 cookie= 2024-04-16T06:05:21.551 INFO:tasks.workunit.client.0.smithi203.stdout: client.4189 expiration=0.000000 addr=v1:172.21.15.203:0/2355805992 cookie= 2024-04-16T06:05:21.563 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:21.563 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: shared 2024-04-16T06:05:21.563 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:21.563 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:21.563 INFO:tasks.workunit.client.0.smithi203.stdout: client.4189 expiration=0.000000 addr=v1:172.21.15.203:0/2355805992 cookie= 2024-04-16T06:05:22.217 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] ClsLock.TestMultiLocking (2895 ms) 2024-04-16T06:05:22.217 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] ClsLock.TestMeta 2024-04-16T06:05:23.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2024-04-16T06:05:24.586 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:24.587 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: shared 2024-04-16T06:05:24.587 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:24.587 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:24.587 INFO:tasks.workunit.client.0.smithi203.stdout: client.4195 expiration=0.000000 addr=v1:172.21.15.203:0/1698571206 cookie= 2024-04-16T06:05:24.613 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:24.613 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: exclusive 2024-04-16T06:05:24.614 INFO:tasks.workunit.client.0.smithi203.stdout: tag: new_tag 2024-04-16T06:05:24.614 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:24.614 INFO:tasks.workunit.client.0.smithi203.stdout: client.4192 expiration=0.000000 addr=v1:172.21.15.203:0/4224362534 cookie= 2024-04-16T06:05:25.234 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] ClsLock.TestMeta (3017 ms) 2024-04-16T06:05:25.235 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] ClsLock.TestCookie 2024-04-16T06:05:27.560 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:27.560 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: exclusive 2024-04-16T06:05:27.561 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:27.561 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout:lock: mylock 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout: lock_type: shared 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout: tag: 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout: lockers: 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout: client.4197 expiration=0.000000 addr=v1:172.21.15.203:0/2261776803 cookie= 2024-04-16T06:05:27.599 INFO:tasks.workunit.client.0.smithi203.stdout: client.4197 expiration=0.000000 addr=v1:172.21.15.203:0/2261776803 cookie=new cookie 2024-04-16T06:05:28.245 INFO:tasks.workunit.client.0.smithi203.stdout:[ OK ] ClsLock.TestCookie (3011 ms) 2024-04-16T06:05:28.245 INFO:tasks.workunit.client.0.smithi203.stdout:[ RUN ] ClsLock.TestMultipleLocks 2024-04-16T06:05:28.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2024-04-16T06:05:34.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-16T06:05:34.125 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2024-04-16T06:05:39.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-04-16T06:05:39.428 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-04-16T06:05:44.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-04-16T06:05:44.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2024-04-16T06:05:50.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-04-16T06:05:50.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~102s 2024-04-16T06:05:55.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~21s 2024-04-16T06:05:55.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~108s 2024-04-16T06:06:00.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-04-16T06:06:00.639 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~113s 2024-04-16T06:06:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-04-16T06:06:05.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~118s 2024-04-16T06:06:11.243 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~37s 2024-04-16T06:06:11.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~124s 2024-04-16T06:06:16.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~42s 2024-04-16T06:06:16.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~129s 2024-04-16T06:06:21.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-04-16T06:06:21.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~134s 2024-04-16T06:06:27.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~53s 2024-04-16T06:06:27.152 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~139s 2024-04-16T06:06:32.002 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:32.004+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:32.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~58s 2024-04-16T06:06:32.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~145s 2024-04-16T06:06:37.002 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:37.005+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:37.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2024-04-16T06:06:37.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~150s 2024-04-16T06:06:42.003 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:42.005+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:43.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2024-04-16T06:06:43.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~155s 2024-04-16T06:06:47.004 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:47.005+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:48.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~74s 2024-04-16T06:06:48.363 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~161s 2024-04-16T06:06:52.005 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:52.007+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:53.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-04-16T06:06:53.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~166s 2024-04-16T06:06:57.006 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:06:57.008+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:06:58.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-04-16T06:06:58.967 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~171s 2024-04-16T06:07:02.006 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:02.009+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:04.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~90s 2024-04-16T06:07:04.269 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~177s 2024-04-16T06:07:07.007 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:07.009+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:09.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~95s 2024-04-16T06:07:09.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~182s 2024-04-16T06:07:12.009 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:12.011+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:14.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-04-16T06:07:14.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~187s 2024-04-16T06:07:17.010 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:17.012+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:20.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~106s 2024-04-16T06:07:20.177 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~193s 2024-04-16T06:07:22.010 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:22.012+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:25.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~111s 2024-04-16T06:07:25.480 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~198s 2024-04-16T06:07:27.011 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:27.012+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:30.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2024-04-16T06:07:30.782 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~203s 2024-04-16T06:07:32.012 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:32.014+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:36.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2024-04-16T06:07:36.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~208s 2024-04-16T06:07:37.012 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:37.015+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:41.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2024-04-16T06:07:41.387 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~214s 2024-04-16T06:07:42.013 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:42.015+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:46.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2024-04-16T06:07:46.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~219s 2024-04-16T06:07:47.014 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:47.016+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:51.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2024-04-16T06:07:51.992 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~224s 2024-04-16T06:07:52.015 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:52.017+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:57.016 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:07:57.018+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:07:57.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2024-04-16T06:07:57.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~230s 2024-04-16T06:08:02.016 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:02.019+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:02.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~148s 2024-04-16T06:08:02.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~235s 2024-04-16T06:08:07.017 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:07.018+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:07.900 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2024-04-16T06:08:07.901 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~240s 2024-04-16T06:08:12.018 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:12.019+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:13.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~159s 2024-04-16T06:08:13.203 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~246s 2024-04-16T06:08:17.019 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:17.021+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:18.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~164s 2024-04-16T06:08:18.506 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~251s 2024-04-16T06:08:22.021 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:22.022+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:23.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-04-16T06:08:23.808 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~256s 2024-04-16T06:08:27.020 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:27.022+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:29.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-04-16T06:08:29.111 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~261s 2024-04-16T06:08:32.021 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:32.023+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:34.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~180s 2024-04-16T06:08:34.413 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~267s 2024-04-16T06:08:37.022 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:37.024+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:39.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2024-04-16T06:08:39.716 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~272s 2024-04-16T06:08:42.023 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:42.025+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:45.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2024-04-16T06:08:45.018 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~277s 2024-04-16T06:08:47.023 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:47.025+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:50.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~196s 2024-04-16T06:08:50.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~283s 2024-04-16T06:08:52.025 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:52.026+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:08:55.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~201s 2024-04-16T06:08:55.623 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~288s 2024-04-16T06:08:57.025 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:08:57.027+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:09:00.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2024-04-16T06:09:00.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~293s 2024-04-16T06:09:02.027 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:09:02.028+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:09:06.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~212s 2024-04-16T06:09:06.229 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~299s 2024-04-16T06:09:07.027 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:09:07.029+0000 7f7f637f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.203:6803/2317714937 for 30sec e34 v34) 2024-04-16T06:09:11.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~217s 2024-04-16T06:09:11.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~304s 2024-04-16T06:09:11.532 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T06:09:11.835 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-16T06:09:11.835 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T06:09:11.836 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T06:09:11.836 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:09:11.839+0000 7f7f667ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 31839) UID: 0 2024-04-16T06:09:11.836 INFO:tasks.ceph.mon.a.smithi203.stderr:2024-04-16T06:09:11.839+0000 7f7f667ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T06:09:11.837 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-04-16T06:09:11.839+0000 7f4436bf7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 31841) UID: 0 2024-04-16T09:05:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T09:05:19.330 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh', 'cls/test_cls_lock.sh', 'cls/test_cls_journal.sh'] on client.0... 2024-04-16T09:05:19.331 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-16T09:05:19.611 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_lock.sh) on smithi203 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh' 2024-04-16T09:05:19.813 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e0e1d88608a848cd9dc3b4b2750e7c13 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_lock.sh) on smithi203 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh' 2024-04-16T09:05:19.815 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T09:05:19.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T09:05:19.825 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T09:05:19.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-04-16T09:05:19.937+0000 7fe4d891d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T09:07:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T09:07:19.864 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:07:19.876 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T09:07:19.876 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T09:07:19.876 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T09:07:19.876 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:07:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:19.879 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2024-04-16T09:07:19.888 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T09:07:19.889 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T09:07:19.889 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:07:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:19.889 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-16T09:07:19.898 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T09:07:19.898 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T09:07:19.898 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:07:19.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:19.899 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-04-16T09:07:19.910 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T09:07:19.910 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T09:07:19.910 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T09:07:19.910 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:07:19.910 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T09:07:19.911 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T09:07:19.911 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T09:07:19.911 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:07:19.911 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T09:07:19.911 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T09:07:19.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-16T09:07:19.963 INFO:teuthology.orchestra.run.smithi203.stdout:2024-04-16T06:04:28.042662+0000 mon.a (mon.0) 129 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T09:07:19.964 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-16T09:07:19.964 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T09:07:19.964 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T09:07:20.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T09:07:20.064 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T09:07:20.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T09:07:20.122 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T09:07:20.190 INFO:tasks.ceph:Archiving mon data... 2024-04-16T09:07:20.192 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938/data/mon.a.tgz 2024-04-16T09:07:20.192 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-04-16T09:07:20.211 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.8gEm2eMAFO 2024-04-16T09:07:20.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8gEm2eMAFO 2024-04-16T09:07:20.333 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.8gEm2eMAFO 2024-04-16T09:07:20.420 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.8gEm2eMAFO is 227KB 2024-04-16T09:07:20.512 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.8gEm2eMAFO 2024-04-16T09:07:20.527 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T09:07:20.527 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T09:07:20.654 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T09:07:20.654 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T09:07:20.656 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938/remote/smithi203/crash 2024-04-16T09:07:20.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T09:07:20.722 INFO:tasks.ceph:Compressing logs... 2024-04-16T09:07:20.722 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T09:07:20.803 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28612.log 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34552.log 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.tmp-client.admin.28612.log: gzip -5 66.9% --verbose -- -- replaced with /var/log/ceph/ceph.tmp-client.admin.28612.log.gz 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr: /var/log/ceph/ceph-client.admin.32041.log 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34552.log: /var/log/ceph/ceph-client.admin.32041.log: 84.0% -- replaced with /var/log/ceph/ceph-client.admin.34552.log.gz 2024-04-16T09:07:20.804 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32279.log 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32041.log.gz 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33723.log 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.32279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33931.log 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32279.log.gz 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33850.log 2024-04-16T09:07:20.805 INFO:teuthology.orchestra.run.smithi203.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33723.log.gz 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33931.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33931.log.gz 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33651.log 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33850.log.gz 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34186.log 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr: 86.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T09:07:20.806 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33651.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33651.log.gz 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34186.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.34186.log.gz 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31931.log 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32081.log 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.31931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33808.log 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31931.log.gz 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.32081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33766.log 2024-04-16T09:07:20.807 INFO:teuthology.orchestra.run.smithi203.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32081.log.gz 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33808.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33808.log.gz 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33595.log 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33766.log.gz 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34058.log 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2024-04-16T09:07:20.808 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34144.log 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33595.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33595.log.gz 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31843.log 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34058.log.gz 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34144.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34144.log.gz 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33930.log 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.31843.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31843.log.gz 2024-04-16T09:07:20.809 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32309.log 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34448.log 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33930.log.gz 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.32309.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32309.log.gz 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34448.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T09:07:20.810 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34448.log.gz 2024-04-16T09:07:20.811 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T09:07:20.811 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34228.log 2024-04-16T09:07:20.811 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32121.log 2024-04-16T09:07:20.811 INFO:teuthology.orchestra.run.smithi203.stderr: 89.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T09:07:20.811 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34228.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34228.log.gz 2024-04-16T09:07:20.821 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34057.log 2024-04-16T09:07:20.821 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.32121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33932.log 2024-04-16T09:07:20.822 INFO:teuthology.orchestra.run.smithi203.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32121.log.gz 2024-04-16T09:07:20.822 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34057.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34057.log.gz 2024-04-16T09:07:20.822 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.33932.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33932.log.gz 2024-04-16T09:07:20.848 INFO:teuthology.orchestra.run.smithi203.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T09:07:20.853 INFO:teuthology.orchestra.run.smithi203.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T09:07:20.945 INFO:teuthology.orchestra.run.smithi203.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T09:07:20.972 INFO:teuthology.orchestra.run.smithi203.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T09:07:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 96.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T09:07:21.313 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-04-16T09:07:21.313 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.535s 2024-04-16T09:07:21.314 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m0.914s 2024-04-16T09:07:21.314 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.082s 2024-04-16T09:07:21.314 INFO:tasks.ceph:Archiving logs... 2024-04-16T09:07:21.314 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938/remote/smithi203/log 2024-04-16T09:07:21.316 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T09:07:21.559 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:07:21.568 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T09:07:21.577 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:07:21.581 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T09:07:21.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T09:07:21.639 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T09:07:21.646 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T09:07:21.646 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T09:07:21.666 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv01.front.sepia.ceph.com 3 8 377 118 +1298us[+1298us] +/- 48ms 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 8 377 251 -1371us[-1362us] +/- 71ms 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -119us[ -119us] +/- 69ms 2024-04-16T09:07:21.671 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T09:07:21.672 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T09:07:21.697 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T09:07:21.697 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T09:07:21.747 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T09:07:21.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T09:07:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:21.844 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T09:07:21.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T09:07:21.881 INFO:teuthology.task.internal:Duration was 11567.545265 seconds 2024-04-16T09:07:21.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T09:07:21.934 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T09:07:21.934 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T09:07:21.979 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T09:07:22.369 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T09:07:22.369 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-04-16T09:07:22.370 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T09:07:22.390 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T09:07:22.391 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T09:07:22.461 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T09:07:22.461 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T09:07:22.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T09:07:22.617 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T09:07:22.617 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T09:07:22.645 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T09:07:22.665 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T09:07:22.713 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-04-16T09:07:22.758 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T09:07:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:22.814 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T09:07:22.823 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T09:07:22.823 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938/remote/smithi203 2024-04-16T09:07:22.824 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T09:07:22.891 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T09:07:22.891 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T09:07:22.941 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T09:07:22.950 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T09:07:22.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T09:07:22.985 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T09:07:22.985 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T09:07:23.000 INFO:teuthology.orchestra.run.smithi203.stdout: 265317 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 09:07 /home/ubuntu/cephtest 2024-04-16T09:07:23.000 INFO:teuthology.orchestra.run.smithi203.stdout: 403682 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 06:02 /home/ubuntu/cephtest/mnt.0 2024-04-16T09:07:23.000 INFO:teuthology.orchestra.run.smithi203.stdout: 403683 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 06:05 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-16T09:07:23.000 INFO:teuthology.orchestra.run.smithi203.stdout: 418052 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 06:05 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-16T09:07:23.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:07:23.001 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T09:07:23.002 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi203 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T09:07:23.014 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T09:07:23.031 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T09:07:23.083 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi203 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:07:23.083 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T09:07:23.111 DEBUG:teuthology.nuke:shortname: smithi203 2024-04-16T09:07:23.112 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T09:07:23.136 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657938', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:46:05.980831', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFdcU0XPAzrvQBbtautmoCISNbXxgmRUfXjJ9U2mrm0GFLwrq4AlHFIDFIxMsK9mt8RIXf4akKLJpLrdgG7Slh0='} 2024-04-16T09:07:23.189 INFO:teuthology.orchestra.console.smithi203:Power off 2024-04-16T09:07:23.189 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T09:07:23.212 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-04-16T09:07:23.212 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:07:23.234 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-16T09:07:27.235 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:07:27.344 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-16T09:07:31.349 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:07:31.458 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-04-16T09:07:35.462 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:07:35.572 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-04-16T09:07:35.673 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-04-16T09:07:35.778 INFO:teuthology.run:Summary data: description: crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} duration: 11567.545264720917 failure_reason: 'Command failed (workunit test cls/test_cls_lock.sh) on smithi203 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh''' flavor: crimson owner: scheduled_nmordech@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e0e1d88608a848cd9dc3b4b2750e7c13 status: fail success: false 2024-04-16T09:07:35.779 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T09:07:35.858 INFO:teuthology.run:FAIL