2024-04-16T05:51:04.705 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-16T05:51:04.706 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937 --owner scheduled_nmordech@teuthology --description crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} --name nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi -- /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937/orig.config.yaml 2024-04-16T05:51:04.735 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T05:51:04.811 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937 branch: wip-nitzan-seastore-re-enable-gate-debug-flag description: crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} email: null first_in_suite: false job_id: '7657937' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-nitzan-seastore-re-enable-gate-debug-flag ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug ms: 1 debug osd: 20 osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-1 osd.1: crimson seastar cpu cores: 2-3 osd.2: crimson seastar cpu cores: 4-5 flavor: crimson log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_APP_NOT_ENABLED\) - is full \(reached quota - \(POOL_FULL\) sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: {} install: ceph: flavor: crimson sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: wip-nitzan-seastore-re-enable-gate-debug-flag env: CRIMSON_COMPAT: '1' sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 9883 sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc sleep_before_teardown: 0 suite: crimson-rados:rbd suite_branch: wip-nitzan-seastore-re-enable-gate-debug-flag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc targets: smithi169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaUGh9tzksP6/9SThjKLXdcAk18nfXCKPCFuUruX739LRCFwz928P9bstiBxnORvfJXtxjYL/l+Hyb1IrYXIVs= tasks: - install: null - ceph: conf: mon: mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug monc: 20 flavor: crimson - workunit: clients: client.0: - rbd/crimson/test_crimson_librbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-16_05:45:04 tube: smithi user: nmordech verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-16T05:51:04.811 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa; will attempt to use it 2024-04-16T05:51:04.812 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks 2024-04-16T05:51:04.812 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-16T05:51:04.813 INFO:teuthology.task.internal:Checking packages... 2024-04-16T05:51:04.837 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc' 2024-04-16T05:51:04.837 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-16T05:51:04.837 INFO:teuthology.packaging:ref: None 2024-04-16T05:51:04.837 INFO:teuthology.packaging:tag: None 2024-04-16T05:51:04.837 INFO:teuthology.packaging:branch: wip-nitzan-seastore-re-enable-gate-debug-flag 2024-04-16T05:51:04.837 INFO:teuthology.packaging:sha1: c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T05:51:04.837 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-nitzan-seastore-re-enable-gate-debug-flag 2024-04-16T05:51:05.168 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3085.gc0a28781 2024-04-16T05:51:05.169 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-16T05:51:05.177 INFO:teuthology.task.internal:no buildpackages task found 2024-04-16T05:51:05.177 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-16T05:51:05.190 INFO:teuthology.task.internal:Saving configuration 2024-04-16T05:51:05.201 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-16T05:51:05.211 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T05:51:05.237 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:46:05.681535', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaUGh9tzksP6/9SThjKLXdcAk18nfXCKPCFuUruX739LRCFwz928P9bstiBxnORvfJXtxjYL/l+Hyb1IrYXIVs='} 2024-04-16T05:51:05.237 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-16T05:51:05.247 INFO:teuthology.task.internal:roles: ubuntu@smithi169.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-16T05:51:05.248 INFO:teuthology.run_tasks:Running task console_log... 2024-04-16T05:51:05.293 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4ae32e5e50>, signals=[15]) 2024-04-16T05:51:05.293 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-16T05:51:05.302 INFO:teuthology.task.internal:Opening connections... 2024-04-16T05:51:05.302 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T05:51:05.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:51:05.395 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-16T05:51:05.405 DEBUG:teuthology.orchestra.run.smithi169:> uname -m 2024-04-16T05:51:05.463 INFO:teuthology.orchestra.run.smithi169.stdout:x86_64 2024-04-16T05:51:05.463 DEBUG:teuthology.orchestra.run.smithi169:> cat /etc/os-release 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:NAME="CentOS Stream" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION="9" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:ID="centos" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:ID_LIKE="rhel fedora" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:VERSION_ID="9" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:PLATFORM_ID="platform:el9" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:ANSI_COLOR="0;31" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:LOGO="fedora-logo-icon" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:HOME_URL="https://centos.org/" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-16T05:51:05.520 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-16T05:51:05.521 INFO:teuthology.orchestra.run.smithi169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-16T05:51:05.521 INFO:teuthology.lock.ops:Updating smithi169.front.sepia.ceph.com on lock server 2024-04-16T05:51:05.547 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-16T05:51:05.559 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-16T05:51:05.571 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-16T05:51:05.572 DEBUG:teuthology.orchestra.run.smithi169:> test '!' -e /home/ubuntu/cephtest 2024-04-16T05:51:05.588 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-16T05:51:05.598 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-16T05:51:05.598 DEBUG:teuthology.orchestra.run.smithi169:> test -z $(ls -A /var/lib/ceph) 2024-04-16T05:51:05.650 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-16T05:51:05.682 INFO:teuthology.run_tasks:Running task kernel... 2024-04-16T05:51:05.696 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-16T05:51:05.697 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-16T05:51:05.697 DEBUG:teuthology.orchestra.run.smithi169:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-16T05:51:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:51:05.713 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-04-16T05:51:05.770 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-361.el9.x86_64 2024-04-16T05:51:05.770 INFO:teuthology.task.kernel:Running kernel on smithi169: 5.14.0-361.el9.x86_64 2024-04-16T05:51:05.770 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum install -y kernel 2024-04-16T05:51:08.283 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-04-16T05:51:10.100 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-04-16T05:51:11.032 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-16T05:51:13.458 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-16T05:51:17.628 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.2 MB 00:00 2024-04-16T05:51:19.101 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-04-16T05:51:20.203 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-16T05:51:25.277 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-04-16T05:51:27.626 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-16T05:51:27.626 INFO:teuthology.orchestra.run.smithi169.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-16T05:51:27.720 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T05:51:27.731 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:51:27.731 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T05:51:27.731 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:51:27.731 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T05:51:27.731 INFO:teuthology.orchestra.run.smithi169.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout:Install 4 Packages 2024-04-16T05:51:27.732 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:51:27.733 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 77 M 2024-04-16T05:51:27.733 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 123 M 2024-04-16T05:51:27.733 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T05:51:27.887 INFO:teuthology.orchestra.run.smithi169.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.8 MB/s | 277 kB 00:00 2024-04-16T05:51:28.371 INFO:teuthology.orchestra.run.smithi169.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-04-16T05:51:29.263 INFO:teuthology.orchestra.run.smithi169.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-04-16T05:51:29.521 INFO:teuthology.orchestra.run.smithi169.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-16T05:51:29.524 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:51:29.524 INFO:teuthology.orchestra.run.smithi169.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-16T05:51:29.525 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T05:51:29.943 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T05:51:29.943 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T05:51:30.796 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T05:51:30.796 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T05:51:33.117 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T05:51:33.709 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-16T05:51:33.788 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:51:36.559 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:51:36.692 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:51:41.421 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:51:41.458 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:51:46.218 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:15.861 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:15.862 INFO:teuthology.orchestra.run.smithi169.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-16T05:52:15.862 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:52:15.905 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:16.309 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:16.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-16T05:52:16.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-16T05:52:16.309 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-16T05:52:16.742 INFO:teuthology.orchestra.run.smithi169.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-16T05:52:16.743 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:52:16.743 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T05:52:17.091 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T05:52:17.213 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:17.213 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T05:52:17.213 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-16T05:52:17.213 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-16T05:52:17.213 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-16T05:52:17.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-16T05:52:17.214 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi169.front.sepia.ceph.com, path=None, version=distro) 2024-04-16T05:52:17.214 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q kernel | sort -rV | head -n 1 2024-04-16T05:52:17.294 INFO:teuthology.orchestra.run.smithi169.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-16T05:52:17.295 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-16T05:52:17.295 DEBUG:teuthology.orchestra.run.smithi169:> sudo rpm -qi grub2-tools 2024-04-16T05:52:17.329 INFO:teuthology.orchestra.run.smithi169.stdout:Name : grub2-tools 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Epoch : 1 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Version : 2.06 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Release : 68.el9 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Architecture: x86_64 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Group : Unspecified 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Size : 8264254 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:License : GPLv3+ 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Packager : builder@centos.org 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:Vendor : CentOS 2024-04-16T05:52:17.330 INFO:teuthology.orchestra.run.smithi169.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:Summary : Support tools for GRUB. 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:Description : 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:hardware devices. 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:52:17.331 INFO:teuthology.orchestra.run.smithi169.stdout:This subpackage provides tools for support of all platforms. 2024-04-16T05:52:17.332 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-16T05:52:17.332 INFO:teuthology.task.kernel:Updating grub on smithi169 to boot 5.14.0-438.el9.x86_64 2024-04-16T05:52:17.332 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-16T05:52:17.946 INFO:teuthology.orchestra.run.smithi169.stderr:Generating grub configuration file ... 2024-04-16T05:52:19.364 INFO:teuthology.orchestra.run.smithi169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-16T05:52:19.415 INFO:teuthology.orchestra.run.smithi169.stderr:done 2024-04-16T05:52:19.416 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-04-16T05:52:19.437 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.EnKsYkHOY2 2024-04-16T05:52:19.437 DEBUG:teuthology.orchestra.run.smithi169:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EnKsYkHOY2 2024-04-16T05:52:19.513 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.EnKsYkHOY2 2024-04-16T05:52:19.674 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.EnKsYkHOY2 is 6KB 2024-04-16T05:52:19.709 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.EnKsYkHOY2 2024-04-16T05:52:19.728 DEBUG:teuthology.orchestra.run.smithi169:> sudo /bin/ls /boot/loader/entries || true 2024-04-16T05:52:19.803 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-16T05:52:19.803 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-16T05:52:19.803 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-16T05:52:19.803 INFO:teuthology.orchestra.run.smithi169.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-16T05:52:19.804 DEBUG:teuthology.orchestra.run.smithi169:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-16T05:52:19.957 DEBUG:teuthology.orchestra.run.smithi169:> sudo shutdown -r now 2024-04-16T05:52:49.979 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-16T05:52:49.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:52:49.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:53:24.511 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-04-16T05:53:33.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:53:33.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:53:51.903 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.169 2024-04-16T05:54:03.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:54:03.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:54:05.540 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-04-16T05:54:06.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:54:06.017 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-16T05:54:06.017 DEBUG:teuthology.orchestra.run.smithi169:> uname -r 2024-04-16T05:54:06.032 INFO:teuthology.orchestra.run.smithi169.stdout:5.14.0-438.el9.x86_64 2024-04-16T05:54:06.032 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-16T05:54:06.032 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-16T05:54:06.032 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-16T05:54:07.033 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-16T05:54:07.070 DEBUG:teuthology.orchestra.run.smithi169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-16T05:54:07.155 INFO:teuthology.orchestra.run.smithi169.stdout:ttyS1 2024-04-16T05:54:07.184 DEBUG:teuthology.parallel:result is None 2024-04-16T05:54:07.184 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-16T05:54:07.192 INFO:teuthology.task.internal:Creating test directory... 2024-04-16T05:54:07.193 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-16T05:54:07.208 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-16T05:54:07.216 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-16T05:54:07.225 INFO:teuthology.task.internal:Creating archive directory... 2024-04-16T05:54:07.225 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-16T05:54:07.275 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-16T05:54:07.283 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-16T05:54:07.283 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-16T05:54:07.357 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:54:07.371 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-16T05:54:07.373 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-16T05:54:07.383 INFO:teuthology.task.internal:Configuring sudo... 2024-04-16T05:54:07.384 DEBUG:teuthology.orchestra.run.smithi169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-16T05:54:07.412 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-16T05:54:07.421 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-16T05:54:07.421 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-16T05:54:07.467 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:54:07.552 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-16T05:54:07.603 DEBUG:teuthology.orchestra.run.smithi169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:54:07.689 DEBUG:teuthology.orchestra.run.smithi169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-16T05:54:07.728 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T05:54:07.729 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-16T05:54:07.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo service rsyslog restart 2024-04-16T05:54:07.882 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T05:54:08.215 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-16T05:54:08.224 INFO:teuthology.task.internal:Starting timer... 2024-04-16T05:54:08.224 INFO:teuthology.run_tasks:Running task pcp... 2024-04-16T05:54:08.252 INFO:teuthology.run_tasks:Running task selinux... 2024-04-16T05:54:08.264 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-16T05:54:08.265 DEBUG:teuthology.orchestra.run.smithi169:> sudo service auditd rotate 2024-04-16T05:54:08.326 INFO:teuthology.orchestra.run.smithi169.stdout:Rotating logs: 2024-04-16T05:54:08.328 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-16T05:54:08.328 DEBUG:teuthology.orchestra.run.smithi169:> /usr/sbin/getenforce 2024-04-16T05:54:08.348 INFO:teuthology.orchestra.run.smithi169.stdout:Permissive 2024-04-16T05:54:08.349 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi169.front.sepia.ceph.com': 'permissive'} 2024-04-16T05:54:08.349 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T05:54:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T05:54:08.419 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-16T05:54:08.420 DEBUG:teuthology.orchestra.run.smithi169:> sudo /usr/sbin/setenforce permissive 2024-04-16T05:54:08.493 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-16T05:54:08.507 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-16T05:54:08.512 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-16T05:54:08.642 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-16T05:54:08.659 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-16T05:54:08.661 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi169.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-16T05:58:31.510 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi169.front.sepia.ceph.com')] 2024-04-16T05:58:31.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:58:31.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-16T05:58:31.607 DEBUG:teuthology.orchestra.run.smithi169:> true 2024-04-16T05:58:31.681 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi169.front.sepia.ceph.com' 2024-04-16T05:58:31.682 INFO:teuthology.run_tasks:Running task clock... 2024-04-16T05:58:31.697 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-16T05:58:31.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-16T05:58:31.698 DEBUG:teuthology.orchestra.run.smithi169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T05:58:31.754 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-16T05:58:31.772 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-16T05:58:31.805 INFO:teuthology.orchestra.run.smithi169.stderr:sudo: ntpd: command not found 2024-04-16T05:58:31.818 INFO:teuthology.orchestra.run.smithi169.stdout:506 Cannot talk to daemon 2024-04-16T05:58:31.836 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-16T05:58:31.853 INFO:teuthology.orchestra.run.smithi169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-16T05:58:31.909 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-04-16T05:58:31.912 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T05:58:31.912 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-04-16T05:58:31.913 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:31.913 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:31.913 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:31.913 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T05:58:31.914 INFO:teuthology.run_tasks:Running task install... 2024-04-16T05:58:31.927 DEBUG:teuthology.task.install:project ceph 2024-04-16T05:58:31.928 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}} 2024-04-16T05:58:31.928 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'} 2024-04-16T05:58:31.928 INFO:teuthology.task.install:Using flavor: crimson 2024-04-16T05:58:31.934 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-16T05:58:31.934 INFO:teuthology.task.install:extra packages: [] 2024-04-16T05:58:31.934 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'tag': None, 'wait_for_package': False} 2024-04-16T05:58:31.935 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T05:58:32.089 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/ 2024-04-16T05:58:32.089 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3085.gc0a28781 2024-04-16T05:58:32.263 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-nitzan-seastore-re-enable-gate-debug-flag/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-16T05:58:32.264 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T05:58:32.264 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-16T05:58:32.293 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-16T05:58:32.294 DEBUG:teuthology.orchestra.run.smithi169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-16T05:58:32.363 DEBUG:teuthology.orchestra.run.smithi169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-16T05:58:32.445 DEBUG:teuthology.orchestra.run.smithi169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-16T05:58:32.513 INFO:teuthology.orchestra.run.smithi169.stdout:check_obsoletes = 1 2024-04-16T05:58:32.514 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum clean all 2024-04-16T05:58:32.754 INFO:teuthology.orchestra.run.smithi169.stdout:74 files removed 2024-04-16T05:58:32.778 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-radosgw 2024-04-16T05:58:33.417 INFO:teuthology.orchestra.run.smithi169.stdout:ceph packages for x86_64 261 kB/s | 87 kB 00:00 2024-04-16T05:58:33.776 INFO:teuthology.orchestra.run.smithi169.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-16T05:58:34.144 INFO:teuthology.orchestra.run.smithi169.stdout:ceph source packages 6.2 kB/s | 2.0 kB 00:00 2024-04-16T05:58:34.571 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-04-16T05:58:35.972 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-04-16T05:58:45.803 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - BaseOS 868 kB/s | 8.0 MB 00:09 2024-04-16T05:58:47.962 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-16T05:58:52.371 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.2 MB 00:00 2024-04-16T05:58:54.014 INFO:teuthology.orchestra.run.smithi169.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 16 kB 00:00 2024-04-16T05:58:56.060 INFO:teuthology.orchestra.run.smithi169.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-16T05:59:01.148 INFO:teuthology.orchestra.run.smithi169.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-04-16T05:59:03.911 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 10 M 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrading: 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: librados2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.4 M 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.7 M 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-16T05:59:03.925 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.9 M 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 19 M 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 25 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 796 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 549 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-16T05:59:03.926 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2 x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.2 M 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 45 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 135 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 159 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 315 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 294 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 99 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-16T05:59:03.927 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout:Installing weak dependencies: 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout:Install 30 Packages 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout:Upgrade 2 Packages 2024-04-16T05:59:03.928 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:03.929 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 59 M 2024-04-16T05:59:03.929 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T05:59:05.375 INFO:teuthology.orchestra.run.smithi169.stdout:(1/32): ceph-radosgw-19.0.0-3085.gc0a28781.el9. 7.9 MB/s | 10 MB 00:01 2024-04-16T05:59:05.417 INFO:teuthology.orchestra.run.smithi169.stdout:(2/32): ceph-selinux-19.0.0-3085.gc0a28781.el9. 606 kB/s | 25 kB 00:00 2024-04-16T05:59:05.542 INFO:teuthology.orchestra.run.smithi169.stdout:(3/32): ceph-base-19.0.0-3085.gc0a28781.el9.x86 4.1 MB/s | 5.9 MB 00:01 2024-04-16T05:59:05.576 INFO:teuthology.orchestra.run.smithi169.stdout:(4/32): libcephfs2-19.0.0-3085.gc0a28781.el9.x8 4.9 MB/s | 796 kB 00:00 2024-04-16T05:59:05.709 INFO:teuthology.orchestra.run.smithi169.stdout:(5/32): libradosstriper1-19.0.0-3085.gc0a28781. 3.2 MB/s | 549 kB 00:00 2024-04-16T05:59:05.893 INFO:teuthology.orchestra.run.smithi169.stdout:(6/32): ceph-common-19.0.0-3085.gc0a28781.el9.x 11 MB/s | 19 MB 00:01 2024-04-16T05:59:05.909 INFO:teuthology.orchestra.run.smithi169.stdout:(7/32): python3-ceph-argparse-19.0.0-3085.gc0a2 225 kB/s | 45 kB 00:00 2024-04-16T05:59:05.960 INFO:teuthology.orchestra.run.smithi169.stdout:(8/32): python3-ceph-common-19.0.0-3085.gc0a287 2.0 MB/s | 135 kB 00:00 2024-04-16T05:59:05.985 INFO:teuthology.orchestra.run.smithi169.stdout:(9/32): python3-cephfs-19.0.0-3085.gc0a28781.el 2.1 MB/s | 159 kB 00:00 2024-04-16T05:59:06.085 INFO:teuthology.orchestra.run.smithi169.stdout:(10/32): librgw2-19.0.0-3085.gc0a28781.el9.x86_ 10 MB/s | 5.2 MB 00:00 2024-04-16T05:59:06.111 INFO:teuthology.orchestra.run.smithi169.stdout:(11/32): python3-rados-19.0.0-3085.gc0a28781.el 2.0 MB/s | 315 kB 00:00 2024-04-16T05:59:06.136 INFO:teuthology.orchestra.run.smithi169.stdout:(12/32): python3-rgw-19.0.0-3085.gc0a28781.el9. 1.9 MB/s | 99 kB 00:00 2024-04-16T05:59:06.170 INFO:teuthology.orchestra.run.smithi169.stdout:(13/32): python3-rbd-19.0.0-3085.gc0a28781.el9. 1.6 MB/s | 294 kB 00:00 2024-04-16T05:59:06.328 INFO:teuthology.orchestra.run.smithi169.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 487 kB/s | 106 kB 00:00 2024-04-16T05:59:06.353 INFO:teuthology.orchestra.run.smithi169.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 328 kB/s | 44 kB 00:00 2024-04-16T05:59:06.379 INFO:teuthology.orchestra.run.smithi169.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 656 kB/s | 159 kB 00:00 2024-04-16T05:59:06.404 INFO:teuthology.orchestra.run.smithi169.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-04-16T05:59:06.429 INFO:teuthology.orchestra.run.smithi169.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-04-16T05:59:06.454 INFO:teuthology.orchestra.run.smithi169.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 553 kB/s | 41 kB 00:00 2024-04-16T05:59:06.480 INFO:teuthology.orchestra.run.smithi169.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-04-16T05:59:06.505 INFO:teuthology.orchestra.run.smithi169.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-04-16T05:59:06.539 INFO:teuthology.orchestra.run.smithi169.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 553 kB/s | 46 kB 00:00 2024-04-16T05:59:06.739 INFO:teuthology.orchestra.run.smithi169.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-04-16T05:59:06.906 INFO:teuthology.orchestra.run.smithi169.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-04-16T05:59:07.006 INFO:teuthology.orchestra.run.smithi169.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 587 kB/s | 309 kB 00:00 2024-04-16T05:59:07.031 INFO:teuthology.orchestra.run.smithi169.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 537 kB/s | 67 kB 00:00 2024-04-16T05:59:07.190 INFO:teuthology.orchestra.run.smithi169.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-16T05:59:07.224 INFO:teuthology.orchestra.run.smithi169.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-04-16T05:59:07.333 INFO:teuthology.orchestra.run.smithi169.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-04-16T05:59:07.433 INFO:teuthology.orchestra.run.smithi169.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-04-16T05:59:08.358 INFO:teuthology.orchestra.run.smithi169.stdout:(31/32): librados2-19.0.0-3085.gc0a28781.el9.x8 3.0 MB/s | 3.4 MB 00:01 2024-04-16T05:59:08.500 INFO:teuthology.orchestra.run.smithi169.stdout:(32/32): librbd1-19.0.0-3085.gc0a28781.el9.x86_ 3.1 MB/s | 3.7 MB 00:01 2024-04-16T05:59:08.504 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:59:08.504 INFO:teuthology.orchestra.run.smithi169.stdout:Total 13 MB/s | 59 MB 00:04 2024-04-16T05:59:08.505 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T05:59:08.542 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T05:59:08.542 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T05:59:09.027 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T05:59:09.027 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T05:59:10.465 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T05:59:11.018 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-16T05:59:11.135 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T05:59:11.210 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T05:59:11.298 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 3/34 2024-04-16T05:59:11.328 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T05:59:11.374 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T05:59:11.415 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-16T05:59:11.441 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-16T05:59:11.467 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-16T05:59:11.490 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T05:59:11.507 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-16T05:59:11.649 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T05:59:11.681 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-16T05:59:12.038 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-16T05:59:12.068 INFO:teuthology.orchestra.run.smithi169.stdout: Upgrading : librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T05:59:12.127 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T05:59:12.170 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9. 12/34 2024-04-16T05:59:12.226 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 13/34 2024-04-16T05:59:12.258 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 14/34 2024-04-16T05:59:12.339 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-16T05:59:12.364 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 16/34 2024-04-16T05:59:12.452 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 16/34 2024-04-16T05:59:12.487 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-16T05:59:12.597 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-16T05:59:12.637 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-16T05:59:12.679 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-16T05:59:12.712 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-16T05:59:12.740 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-16T05:59:12.772 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-16T05:59:12.810 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-16T05:59:13.378 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-16T05:59:13.412 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 26/34 2024-04-16T05:59:13.483 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 26/34 2024-04-16T05:59:13.552 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 27/34 2024-04-16T05:59:13.581 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x8 28/34 2024-04-16T05:59:15.951 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:15.986 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:16.675 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 29/34 2024-04-16T05:59:16.714 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 30/34 2024-04-16T05:59:16.789 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 30/34 2024-04-16T05:59:16.789 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-16T05:59:16.789 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-16T05:59:16.789 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:16.824 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T05:59:26.821 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T05:59:26.821 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /sys 2024-04-16T05:59:26.821 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /proc 2024-04-16T05:59:26.821 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /mnt 2024-04-16T05:59:26.822 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /var/tmp 2024-04-16T05:59:26.822 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /home 2024-04-16T05:59:26.822 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /root 2024-04-16T05:59:26.822 INFO:teuthology.orchestra.run.smithi169.stdout:skipping the directory /tmp 2024-04-16T05:59:26.822 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:27.905 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 32/34 2024-04-16T05:59:27.930 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 32/34 2024-04-16T05:59:27.930 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T05:59:27.930 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-16T05:59:27.930 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:59:27.931 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-16T05:59:27.931 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:27.932 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T05:59:27.998 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-16T05:59:27.999 INFO:teuthology.orchestra.run.smithi169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 1/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 3/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 4/34 2024-04-16T05:59:29.259 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 5/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_6 6/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 7/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9. 8/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x8 9/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 10/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 11/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 12/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 13/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-16T05:59:29.260 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-16T05:59:29.261 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 31/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-16T05:59:29.262 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 33/34 2024-04-16T05:59:30.318 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-16T05:59:30.318 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:30.318 INFO:teuthology.orchestra.run.smithi169.stdout:Upgraded: 2024-04-16T05:59:30.318 INFO:teuthology.orchestra.run.smithi169.stdout: librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.318 INFO:teuthology.orchestra.run.smithi169.stdout: librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-radosgw-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-selinux-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-16T05:59:30.319 INFO:teuthology.orchestra.run.smithi169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: libradosstriper1-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: librgw2-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-argparse-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ceph-common-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T05:59:30.320 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: re2-1:20211101-3.el9.x86_64 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:30.321 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T05:59:30.667 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-test 2024-04-16T05:59:31.247 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:00:30 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T05:59:31.406 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 84 M 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 174 k 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-04-16T05:59:31.407 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout:Install 11 Packages 2024-04-16T05:59:31.408 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T05:59:31.409 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 87 M 2024-04-16T05:59:31.409 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 814 M 2024-04-16T05:59:31.409 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T05:59:31.999 INFO:teuthology.orchestra.run.smithi169.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.6 MB/s | 217 kB 00:00 2024-04-16T05:59:32.074 INFO:teuthology.orchestra.run.smithi169.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-16T05:59:32.116 INFO:teuthology.orchestra.run.smithi169.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.2 MB/s | 300 kB 00:00 2024-04-16T05:59:32.141 INFO:teuthology.orchestra.run.smithi169.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-04-16T05:59:32.175 INFO:teuthology.orchestra.run.smithi169.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-04-16T05:59:32.259 INFO:teuthology.orchestra.run.smithi169.stdout:(6/11): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-04-16T05:59:32.443 INFO:teuthology.orchestra.run.smithi169.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.5 MB/s | 469 kB 00:00 2024-04-16T05:59:32.493 INFO:teuthology.orchestra.run.smithi169.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-16T05:59:32.568 INFO:teuthology.orchestra.run.smithi169.stdout:(9/11): libcephsqlite-19.0.0-3085.gc0a28781.el9 247 kB/s | 174 kB 00:00 2024-04-16T05:59:32.726 INFO:teuthology.orchestra.run.smithi169.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 521 kB/s | 122 kB 00:00 2024-04-16T05:59:45.351 INFO:teuthology.orchestra.run.smithi169.stdout:(11/11): ceph-test-19.0.0-3085.gc0a28781.el9.x8 6.3 MB/s | 84 MB 00:13 2024-04-16T05:59:45.354 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T05:59:45.354 INFO:teuthology.orchestra.run.smithi169.stdout:Total 6.2 MB/s | 87 MB 00:13 2024-04-16T05:59:45.397 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T05:59:45.421 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T05:59:45.421 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T05:59:46.053 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T05:59:46.053 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T05:59:47.055 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T05:59:47.115 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-04-16T05:59:47.160 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-04-16T05:59:47.185 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T05:59:47.251 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-04-16T05:59:47.285 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-04-16T05:59:47.329 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-04-16T05:59:47.374 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-04-16T05:59:47.409 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-04-16T05:59:47.462 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-04-16T05:59:47.510 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-04-16T05:59:47.532 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 10/11 2024-04-16T05:59:58.855 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 10/11 2024-04-16T05:59:58.931 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 11/11 2024-04-16T06:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 11/11 2024-04-16T06:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 1/11 2024-04-16T06:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 2/11 2024-04-16T06:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-04-16T06:00:05.216 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-04-16T06:00:05.217 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-test-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout: jq-1.6-16.el9.x86_64 2024-04-16T06:00:05.579 INFO:teuthology.orchestra.run.smithi169.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: libcephsqlite-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:05.580 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:05.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph 2024-04-16T06:00:06.479 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:05 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:06.632 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:06.634 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-04-16T06:00:06.634 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 6.5 k 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 2.2 M 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 1.6 M 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 249 k 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 5.0 M 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 71 M 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-16T06:00:06.635 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-16T06:00:06.636 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-16T06:00:06.637 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout:========================================================================================= 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout:Install 36 Packages 2024-04-16T06:00:06.638 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:06.639 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 84 M 2024-04-16T06:00:06.639 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 922 M 2024-04-16T06:00:06.639 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:00:07.840 INFO:teuthology.orchestra.run.smithi169.stdout:(1/36): ceph-19.0.0-3085.gc0a28781.el9.x86_64.r 20 kB/s | 6.5 kB 00:00 2024-04-16T06:00:08.532 INFO:teuthology.orchestra.run.smithi169.stdout:(2/36): ceph-mgr-19.0.0-3085.gc0a28781.el9.x86_ 1.5 MB/s | 1.6 MB 00:01 2024-04-16T06:00:09.282 INFO:teuthology.orchestra.run.smithi169.stdout:(3/36): ceph-mon-19.0.0-3085.gc0a28781.el9.x86_ 3.5 MB/s | 5.0 MB 00:01 2024-04-16T06:00:09.333 INFO:teuthology.orchestra.run.smithi169.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3085.gc0a2 4.9 MB/s | 249 kB 00:00 2024-04-16T06:00:09.449 INFO:teuthology.orchestra.run.smithi169.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-04-16T06:00:09.492 INFO:teuthology.orchestra.run.smithi169.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-04-16T06:00:09.569 INFO:teuthology.orchestra.run.smithi169.stdout:(7/36): ceph-mds-19.0.0-3085.gc0a28781.el9.x86_ 1.1 MB/s | 2.2 MB 00:02 2024-04-16T06:00:09.592 INFO:teuthology.orchestra.run.smithi169.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 316 kB/s | 32 kB 00:00 2024-04-16T06:00:09.609 INFO:teuthology.orchestra.run.smithi169.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 47 kB 00:00 2024-04-16T06:00:09.626 INFO:teuthology.orchestra.run.smithi169.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-04-16T06:00:09.651 INFO:teuthology.orchestra.run.smithi169.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-04-16T06:00:09.701 INFO:teuthology.orchestra.run.smithi169.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-04-16T06:00:09.726 INFO:teuthology.orchestra.run.smithi169.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-04-16T06:00:09.752 INFO:teuthology.orchestra.run.smithi169.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-04-16T06:00:09.777 INFO:teuthology.orchestra.run.smithi169.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-04-16T06:00:09.869 INFO:teuthology.orchestra.run.smithi169.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-04-16T06:00:10.219 INFO:teuthology.orchestra.run.smithi169.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 267 kB/s | 125 kB 00:00 2024-04-16T06:00:10.244 INFO:teuthology.orchestra.run.smithi169.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 403 kB/s | 151 kB 00:00 2024-04-16T06:00:10.386 INFO:teuthology.orchestra.run.smithi169.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-04-16T06:00:10.487 INFO:teuthology.orchestra.run.smithi169.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-04-16T06:00:10.520 INFO:teuthology.orchestra.run.smithi169.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.6 MB/s | 358 kB 00:00 2024-04-16T06:00:10.587 INFO:teuthology.orchestra.run.smithi169.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-04-16T06:00:10.612 INFO:teuthology.orchestra.run.smithi169.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-04-16T06:00:10.673 INFO:teuthology.orchestra.run.smithi169.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 277 kB/s | 23 kB 00:00 2024-04-16T06:00:10.696 INFO:teuthology.orchestra.run.smithi169.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 232 kB/s | 19 kB 00:00 2024-04-16T06:00:10.755 INFO:teuthology.orchestra.run.smithi169.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 236 kB/s | 20 kB 00:00 2024-04-16T06:00:10.797 INFO:teuthology.orchestra.run.smithi169.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-04-16T06:00:10.889 INFO:teuthology.orchestra.run.smithi169.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-04-16T06:00:10.915 INFO:teuthology.orchestra.run.smithi169.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 496 kB/s | 79 kB 00:00 2024-04-16T06:00:10.973 INFO:teuthology.orchestra.run.smithi169.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-04-16T06:00:11.015 INFO:teuthology.orchestra.run.smithi169.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 899 kB/s | 90 kB 00:00 2024-04-16T06:00:11.049 INFO:teuthology.orchestra.run.smithi169.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2024-04-16T06:00:11.149 INFO:teuthology.orchestra.run.smithi169.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-04-16T06:00:11.174 INFO:teuthology.orchestra.run.smithi169.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-04-16T06:00:11.208 INFO:teuthology.orchestra.run.smithi169.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-04-16T06:00:21.074 INFO:teuthology.orchestra.run.smithi169.stdout:(36/36): ceph-osd-19.0.0-3085.gc0a28781.el9.x86 5.7 MB/s | 71 MB 00:12 2024-04-16T06:00:21.080 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:21.080 INFO:teuthology.orchestra.run.smithi169.stdout:Total 5.8 MB/s | 84 MB 00:14 2024-04-16T06:00:21.106 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:00:21.150 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:00:21.150 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:00:21.890 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:00:21.891 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:00:23.474 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:00:23.542 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-16T06:00:23.582 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-16T06:00:23.612 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-16T06:00:23.684 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-16T06:00:23.727 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-16T06:00:23.760 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-16T06:00:23.852 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-16T06:00:23.931 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-16T06:00:23.971 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-16T06:00:24.044 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-16T06:00:24.105 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-04-16T06:00:24.143 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-04-16T06:00:24.193 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-04-16T06:00:24.264 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-04-16T06:00:24.449 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-04-16T06:00:24.497 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-04-16T06:00:24.530 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-04-16T06:00:24.579 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-04-16T06:00:24.609 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-04-16T06:00:24.685 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-04-16T06:00:24.719 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T06:00:24.753 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-04-16T06:00:24.793 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-04-16T06:00:24.828 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-04-16T06:00:24.862 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-04-16T06:00:24.953 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-04-16T06:00:25.001 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-04-16T06:00:25.075 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-04-16T06:00:25.212 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-04-16T06:00:25.339 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T06:00:25.535 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9. 31/36 2024-04-16T06:00:25.565 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 32/36 2024-04-16T06:00:25.593 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 32/36 2024-04-16T06:00:25.594 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:25.594 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-16T06:00:25.594 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T06:00:25.594 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-16T06:00:25.594 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:36.406 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 33/36 2024-04-16T06:00:36.434 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 33/36 2024-04-16T06:00:36.434 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:36.435 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-16T06:00:36.435 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T06:00:36.435 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-16T06:00:36.435 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:36.992 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 34/36 2024-04-16T06:00:37.018 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 34/36 2024-04-16T06:00:37.019 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:37.019 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-16T06:00:37.019 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T06:00:37.019 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-16T06:00:37.019 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:37.266 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 35/36 2024-04-16T06:00:37.292 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 35/36 2024-04-16T06:00:37.292 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:37.293 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-16T06:00:37.293 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T06:00:37.293 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-16T06:00:37.293 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:37.357 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 36/36 2024-04-16T06:00:44.188 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 36/36 2024-04-16T06:00:44.188 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 1/36 2024-04-16T06:00:44.188 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 2/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 3/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 4/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 5/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9. 6/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-04-16T06:00:44.189 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-16T06:00:44.190 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-16T06:00:44.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-16T06:00:44.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-16T06:00:44.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-16T06:00:44.191 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-16T06:00:45.078 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-16T06:00:45.078 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mds-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-modules-core-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mon-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-osd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: lua-5.4.4-4.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-16T06:00:45.079 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-ply-3.11-14.el9.noarch 2024-04-16T06:00:45.080 INFO:teuthology.orchestra.run.smithi169.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:45.081 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:45.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-base 2024-04-16T06:00:45.916 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:44 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:46.015 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-base-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:00:46.067 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:46.068 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:00:46.068 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:46.136 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install cephadm 2024-04-16T06:00:46.714 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:45 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:46.867 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 762 k 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:46.868 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:00:46.869 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:46.869 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 762 k 2024-04-16T06:00:46.869 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 768 k 2024-04-16T06:00:46.869 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:00:47.141 INFO:teuthology.orchestra.run.smithi169.stdout:cephadm-19.0.0-3085.gc0a28781.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-04-16T06:00:47.142 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:47.142 INFO:teuthology.orchestra.run.smithi169.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-04-16T06:00:47.142 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:00:47.147 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:00:47.147 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:00:47.153 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:00:47.153 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:00:47.230 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:00:47.686 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:00:47.763 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:00:48.163 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:00:48.487 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/1 2024-04-16T06:00:48.487 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:48.487 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:00:48.487 INFO:teuthology.orchestra.run.smithi169.stdout: cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:48.488 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:48.488 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:48.708 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-immutable-object-cache 2024-04-16T06:00:49.294 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:48 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:49.445 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:49.446 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:49.446 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-04-16T06:00:49.446 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:49.446 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:00:49.446 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 157 k 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 157 k 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 640 k 2024-04-16T06:00:49.447 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:00:49.797 INFO:teuthology.orchestra.run.smithi169.stdout:ceph-immutable-object-cache-19.0.0-3085.gc0a287 448 kB/s | 157 kB 00:00 2024-04-16T06:00:49.798 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:49.798 INFO:teuthology.orchestra.run.smithi169.stdout:Total 446 kB/s | 157 kB 00:00 2024-04-16T06:00:49.798 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:00:49.807 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:00:49.808 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:00:49.865 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:00:49.866 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:00:50.008 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:00:50.082 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:00:50.106 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:00:50.106 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:00:50.106 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-16T06:00:50.106 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781. 1/1 2024-04-16T06:00:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:51.051 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:00:51.051 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-immutable-object-cache-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:00:51.052 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:51.052 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:51.270 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr 2024-04-16T06:00:51.863 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:50 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:51.962 INFO:teuthology.orchestra.run.smithi169.stdout:Package ceph-mgr-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:00:52.017 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:52.018 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:00:52.018 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:52.074 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-dashboard 2024-04-16T06:00:52.656 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:51 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:52.807 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 3.5 M 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 34 k 2024-04-16T06:00:52.808 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 16 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout:====================================================================================== 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout:Install 11 Packages 2024-04-16T06:00:52.809 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:52.810 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 7.6 M 2024-04-16T06:00:52.810 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 100 M 2024-04-16T06:00:52.810 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:00:53.593 INFO:teuthology.orchestra.run.smithi169.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3085.gc0a 51 kB/s | 16 kB 00:00 2024-04-16T06:00:53.635 INFO:teuthology.orchestra.run.smithi169.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3085.gc0 93 kB/s | 34 kB 00:00 2024-04-16T06:00:53.743 INFO:teuthology.orchestra.run.smithi169.stdout:(3/11): ceph-mgr-dashboard-19.0.0-3085.gc0a2878 7.5 MB/s | 3.5 MB 00:00 2024-04-16T06:00:53.860 INFO:teuthology.orchestra.run.smithi169.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-04-16T06:00:53.985 INFO:teuthology.orchestra.run.smithi169.stdout:(5/11): python3-protobuf-3.14.0-13.el9.noarch.r 606 kB/s | 238 kB 00:00 2024-04-16T06:00:54.203 INFO:teuthology.orchestra.run.smithi169.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 57 kB/s | 19 kB 00:00 2024-04-16T06:00:54.519 INFO:teuthology.orchestra.run.smithi169.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 456 kB/s | 144 kB 00:00 2024-04-16T06:00:54.545 INFO:teuthology.orchestra.run.smithi169.stdout:(8/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 685 kB/s | 548 kB 00:00 2024-04-16T06:00:54.629 INFO:teuthology.orchestra.run.smithi169.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 283 kB/s | 31 kB 00:00 2024-04-16T06:00:54.662 INFO:teuthology.orchestra.run.smithi169.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-04-16T06:00:54.796 INFO:teuthology.orchestra.run.smithi169.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.5 MB/s | 2.0 MB 00:00 2024-04-16T06:00:54.799 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:00:54.799 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.8 MB/s | 7.6 MB 00:01 2024-04-16T06:00:54.818 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:00:54.835 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:00:54.835 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:00:55.007 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:00:55.007 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:00:55.495 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:00:55.561 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-16T06:00:55.590 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-16T06:00:55.703 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-16T06:00:55.833 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-16T06:00:55.894 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-16T06:00:56.197 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-16T06:00:56.237 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-16T06:00:56.266 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-16T06:00:56.298 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9 9/11 2024-04-16T06:00:57.351 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el 10/11 2024-04-16T06:00:57.382 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 11/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 11/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el 1/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noa 2/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9 3/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-16T06:00:58.575 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-16T06:00:58.576 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-16T06:00:58.576 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-16T06:00:58.576 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-16T06:00:58.576 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-16T06:00:58.576 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-grafana-dashboards-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-dashboard-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-prometheus-alerts-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-16T06:00:58.926 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:58.927 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:00:59.175 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-16T06:00:59.754 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:01:58 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:00:59.907 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 7.4 M 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-16T06:00:59.908 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout:============================================================================================ 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout:Install 12 Packages 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:00:59.909 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 37 M 2024-04-16T06:00:59.910 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 209 M 2024-04-16T06:00:59.910 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:00.270 INFO:teuthology.orchestra.run.smithi169.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2024-04-16T06:01:00.303 INFO:teuthology.orchestra.run.smithi169.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2024-04-16T06:01:00.345 INFO:teuthology.orchestra.run.smithi169.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 34 kB 00:00 2024-04-16T06:01:00.396 INFO:teuthology.orchestra.run.smithi169.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-04-16T06:01:00.579 INFO:teuthology.orchestra.run.smithi169.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-04-16T06:01:00.646 INFO:teuthology.orchestra.run.smithi169.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-04-16T06:01:00.671 INFO:teuthology.orchestra.run.smithi169.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-04-16T06:01:00.963 INFO:teuthology.orchestra.run.smithi169.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-04-16T06:01:01.230 INFO:teuthology.orchestra.run.smithi169.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-30 6.8 MB/s | 7.4 MB 00:01 2024-04-16T06:01:01.849 INFO:teuthology.orchestra.run.smithi169.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 415 kB/s | 245 kB 00:00 2024-04-16T06:01:01.872 INFO:teuthology.orchestra.run.smithi169.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 215 kB/s | 195 kB 00:00 2024-04-16T06:01:02.456 INFO:teuthology.orchestra.run.smithi169.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 450 kB/s | 803 kB 00:01 2024-04-16T06:01:02.459 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:02.459 INFO:teuthology.orchestra.run.smithi169.stdout:Total 15 MB/s | 37 MB 00:02 2024-04-16T06:01:02.504 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:02.530 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:02.530 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:02.893 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:02.894 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:03.600 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:03.679 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-04-16T06:01:03.710 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-16T06:01:03.811 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-04-16T06:01:03.855 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-04-16T06:01:04.032 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T06:01:04.062 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T06:01:04.138 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-16T06:01:04.803 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-16T06:01:04.924 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-16T06:01:06.188 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-16T06:01:07.201 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-16T06:01:07.237 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 12/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 12/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28 1/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-04-16T06:01:09.909 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-04-16T06:01:09.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-04-16T06:01:09.910 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-16T06:01:10.327 INFO:teuthology.orchestra.run.smithi169.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-16T06:01:10.328 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-16T06:01:10.328 INFO:teuthology.orchestra.run.smithi169.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-16T06:01:10.328 INFO:teuthology.orchestra.run.smithi169.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-16T06:01:10.328 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:10.328 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:10.705 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-rook 2024-04-16T06:01:11.302 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:10 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:11.456 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:11.457 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:11.457 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:01:11.457 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:11.457 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:11.457 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 49 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-16T06:01:11.458 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout:Install 13 Packages 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:11.459 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 2.0 M 2024-04-16T06:01:11.460 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 26 M 2024-04-16T06:01:11.460 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:13.288 INFO:teuthology.orchestra.run.smithi169.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 110 kB/s | 18 kB 00:00 2024-04-16T06:01:13.313 INFO:teuthology.orchestra.run.smithi169.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 129 kB/s | 24 kB 00:00 2024-04-16T06:01:13.363 INFO:teuthology.orchestra.run.smithi169.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-04-16T06:01:13.388 INFO:teuthology.orchestra.run.smithi169.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-16T06:01:13.413 INFO:teuthology.orchestra.run.smithi169.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-04-16T06:01:13.439 INFO:teuthology.orchestra.run.smithi169.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-04-16T06:01:13.489 INFO:teuthology.orchestra.run.smithi169.stdout:(7/13): ceph-mgr-rook-19.0.0-3085.gc0a28781.el9 135 kB/s | 49 kB 00:00 2024-04-16T06:01:13.706 INFO:teuthology.orchestra.run.smithi169.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-04-16T06:01:13.732 INFO:teuthology.orchestra.run.smithi169.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-04-16T06:01:13.882 INFO:teuthology.orchestra.run.smithi169.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 595 kB/s | 203 kB 00:00 2024-04-16T06:01:13.907 INFO:teuthology.orchestra.run.smithi169.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-04-16T06:01:13.957 INFO:teuthology.orchestra.run.smithi169.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-04-16T06:01:13.982 INFO:teuthology.orchestra.run.smithi169.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 897 kB/s | 90 kB 00:00 2024-04-16T06:01:13.986 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:13.987 INFO:teuthology.orchestra.run.smithi169.stdout:Total 829 kB/s | 2.0 MB 00:02 2024-04-16T06:01:13.987 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:14.001 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:14.001 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:14.101 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:14.102 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:14.616 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:14.792 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-16T06:01:14.859 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-16T06:01:14.923 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-16T06:01:14.960 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-16T06:01:15.001 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-16T06:01:15.096 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-16T06:01:15.205 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-16T06:01:15.303 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-16T06:01:16.041 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-16T06:01:16.092 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-16T06:01:16.125 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-16T06:01:16.173 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-16T06:01:16.195 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 13/13 2024-04-16T06:01:17.041 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 13/13 2024-04-16T06:01:17.041 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 1/13 2024-04-16T06:01:17.041 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-16T06:01:17.041 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-16T06:01:17.041 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-16T06:01:17.042 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-16T06:01:17.433 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-16T06:01:17.433 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-rook-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-16T06:01:17.434 INFO:teuthology.orchestra.run.smithi169.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-16T06:01:17.435 INFO:teuthology.orchestra.run.smithi169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-16T06:01:17.435 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:17.435 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:17.688 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-mgr-cephadm 2024-04-16T06:01:18.291 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:17 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:18.446 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:18.446 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:18.446 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 142 k 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:18.447 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout:======================================================================================== 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout:Install 6 Packages 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 6.8 M 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 33 M 2024-04-16T06:01:18.448 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:19.112 INFO:teuthology.orchestra.run.smithi169.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-04-16T06:01:19.271 INFO:teuthology.orchestra.run.smithi169.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-04-16T06:01:19.363 INFO:teuthology.orchestra.run.smithi169.stdout:(3/6): ceph-mgr-cephadm-19.0.0-3085.gc0a28781.e 340 kB/s | 142 kB 00:00 2024-04-16T06:01:19.421 INFO:teuthology.orchestra.run.smithi169.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-04-16T06:01:19.538 INFO:teuthology.orchestra.run.smithi169.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-04-16T06:01:19.563 INFO:teuthology.orchestra.run.smithi169.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 254 kB/s | 51 kB 00:00 2024-04-16T06:01:19.564 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:19.564 INFO:teuthology.orchestra.run.smithi169.stdout:Total 6.1 MB/s | 6.8 MB 00:01 2024-04-16T06:01:19.578 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:19.588 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:19.588 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:19.677 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:19.678 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:19.993 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:20.169 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-16T06:01:20.223 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-16T06:01:20.732 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-16T06:01:20.808 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-16T06:01:20.871 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-16T06:01:20.894 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 6/6 2024-04-16T06:01:21.668 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 6/6 2024-04-16T06:01:21.668 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 1/6 2024-04-16T06:01:21.668 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-16T06:01:21.669 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-16T06:01:21.669 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-16T06:01:21.669 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-mgr-cephadm-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-16T06:01:21.981 INFO:teuthology.orchestra.run.smithi169.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-16T06:01:21.982 INFO:teuthology.orchestra.run.smithi169.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-16T06:01:21.982 INFO:teuthology.orchestra.run.smithi169.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-16T06:01:21.982 INFO:teuthology.orchestra.run.smithi169.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-16T06:01:21.982 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:21.982 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:22.257 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-fuse 2024-04-16T06:01:22.848 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:21 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:23.001 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 870 k 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-16T06:01:23.002 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 949 k 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 4.0 M 2024-04-16T06:01:23.003 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:23.123 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 657 kB/s | 79 kB 00:00 2024-04-16T06:01:23.307 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-fuse-19.0.0-3085.gc0a28781.el9.x86_ 2.8 MB/s | 870 kB 00:00 2024-04-16T06:01:23.308 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:23.308 INFO:teuthology.orchestra.run.smithi169.stdout:Total 3.0 MB/s | 949 kB 00:00 2024-04-16T06:01:23.308 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:23.330 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:23.330 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:23.403 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:23.404 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:23.589 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:23.742 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-16T06:01:23.807 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2/2 2024-04-16T06:01:24.496 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2/2 2024-04-16T06:01:24.496 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/2 2024-04-16T06:01:24.798 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-16T06:01:24.799 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:24.799 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:24.799 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-16T06:01:24.799 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:24.799 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:25.080 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install ceph-volume 2024-04-16T06:01:25.681 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:24 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:25.833 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:25.834 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:25.834 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repository Size 2024-04-16T06:01:25.834 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:25.834 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume noarch 2:19.0.0-3085.gc0a28781.el9 ceph-noarch 264 k 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Installing dependencies: 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Install 2 Packages 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 332 k 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 1.5 M 2024-04-16T06:01:25.835 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:25.945 INFO:teuthology.orchestra.run.smithi169.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 628 kB/s | 69 kB 00:00 2024-04-16T06:01:26.220 INFO:teuthology.orchestra.run.smithi169.stdout:(2/2): ceph-volume-19.0.0-3085.gc0a28781.el9.no 685 kB/s | 264 kB 00:00 2024-04-16T06:01:26.222 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:26.222 INFO:teuthology.orchestra.run.smithi169.stdout:Total 858 kB/s | 332 kB 00:00 2024-04-16T06:01:26.222 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:26.231 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:26.231 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:26.268 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:26.269 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:26.481 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:26.669 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-16T06:01:26.715 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2/2 2024-04-16T06:01:26.731 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2/2 2024-04-16T06:01:26.731 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:01:26.731 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-16T06:01:26.731 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:27.489 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 1/2 2024-04-16T06:01:27.804 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout: ceph-volume-2:19.0.0-3085.gc0a28781.el9.noarch 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:27.805 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:28.101 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados-devel 2024-04-16T06:01:28.709 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:27 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:28.863 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:28.863 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:28.863 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 125 k 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:28.864 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 125 k 2024-04-16T06:01:28.865 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 445 k 2024-04-16T06:01:28.865 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:29.217 INFO:teuthology.orchestra.run.smithi169.stdout:librados-devel-19.0.0-3085.gc0a28781.el9.x86_64 354 kB/s | 125 kB 00:00 2024-04-16T06:01:29.218 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:29.218 INFO:teuthology.orchestra.run.smithi169.stdout:Total 353 kB/s | 125 kB 00:00 2024-04-16T06:01:29.218 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:29.224 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:29.224 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:29.280 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:29.281 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:29.424 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:29.525 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:29.961 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:30.289 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:30.289 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:30.289 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:30.290 INFO:teuthology.orchestra.run.smithi169.stdout: librados-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:30.290 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:30.290 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:30.536 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs2 2024-04-16T06:01:31.132 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:30 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:31.234 INFO:teuthology.orchestra.run.smithi169.stdout:Package libcephfs2-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:31.286 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:31.287 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:31.287 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:31.353 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install libcephfs-devel 2024-04-16T06:01:31.939 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:30 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:32.091 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:32.092 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 32 k 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 32 k 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 142 k 2024-04-16T06:01:32.093 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:32.345 INFO:teuthology.orchestra.run.smithi169.stdout:libcephfs-devel-19.0.0-3085.gc0a28781.el9.x86_6 126 kB/s | 32 kB 00:00 2024-04-16T06:01:32.346 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:32.346 INFO:teuthology.orchestra.run.smithi169.stdout:Total 125 kB/s | 32 kB 00:00 2024-04-16T06:01:32.346 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:32.350 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:32.350 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:32.377 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:32.377 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:32.466 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:32.585 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:32.967 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout: libcephfs-devel-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:33.229 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:33.405 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librados2 2024-04-16T06:01:33.989 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:32 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:34.088 INFO:teuthology.orchestra.run.smithi169.stdout:Package librados2-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:34.140 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:34.141 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:34.141 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:34.215 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install librbd1 2024-04-16T06:01:34.800 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:33 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:34.901 INFO:teuthology.orchestra.run.smithi169.stdout:Package librbd1-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:34.954 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:34.955 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:34.955 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:35.024 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rados 2024-04-16T06:01:35.606 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:34 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:35.704 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rados-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:35.755 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:35.756 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:35.757 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:35.823 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rgw 2024-04-16T06:01:36.399 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:35 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:36.497 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rgw-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:36.548 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:36.549 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:36.550 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:36.611 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-cephfs 2024-04-16T06:01:37.196 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:36 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:37.297 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-cephfs-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:37.350 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:37.351 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:37.351 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:37.411 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install python3-rbd 2024-04-16T06:01:38.003 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:37 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:38.108 INFO:teuthology.orchestra.run.smithi169.stdout:Package python3-rbd-2:19.0.0-3085.gc0a28781.el9.x86_64 is already installed. 2024-04-16T06:01:38.163 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:38.165 INFO:teuthology.orchestra.run.smithi169.stdout:Nothing to do. 2024-04-16T06:01:38.165 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:38.245 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-fuse 2024-04-16T06:01:38.859 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:37 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:39.018 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 90 k 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:39.019 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 90 k 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 313 k 2024-04-16T06:01:39.020 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:39.340 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-fuse-19.0.0-3085.gc0a28781.el9.x86_64.rpm 280 kB/s | 90 kB 00:00 2024-04-16T06:01:39.341 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:39.341 INFO:teuthology.orchestra.run.smithi169.stdout:Total 279 kB/s | 90 kB 00:00 2024-04-16T06:01:39.341 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:39.350 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:39.350 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:39.406 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:39.542 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:39.654 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:40.068 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-fuse-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:40.362 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:40.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-mirror 2024-04-16T06:01:41.164 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:40 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:41.321 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout: Package Arch Version Repo Size 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 3.4 M 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:01:41.322 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:41.323 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 3.4 M 2024-04-16T06:01:41.323 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 29 M 2024-04-16T06:01:41.323 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:41.701 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-mirror-19.0.0-3085.gc0a28781.el9.x86_64.rpm 9.0 MB/s | 3.4 MB 00:00 2024-04-16T06:01:41.702 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:41.702 INFO:teuthology.orchestra.run.smithi169.stdout:Total 8.9 MB/s | 3.4 MB 00:00 2024-04-16T06:01:41.703 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:41.713 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:41.788 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:41.788 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:42.352 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:42.411 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-16T06:01:42.437 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:43.660 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:43.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:43.661 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:43.661 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-mirror-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:43.661 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:43.661 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:43.904 DEBUG:teuthology.orchestra.run.smithi169:> sudo yum -y install rbd-nbd 2024-04-16T06:01:44.509 INFO:teuthology.orchestra.run.smithi169.stdout:Last metadata expiration check: 0:02:43 ago on Tue 16 Apr 2024 05:59:01 AM UTC. 2024-04-16T06:01:44.661 INFO:teuthology.orchestra.run.smithi169.stdout:Dependencies resolved. 2024-04-16T06:01:44.661 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout: Package Architecture Version Repository Size 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Installing: 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd x86_64 2:19.0.0-3085.gc0a28781.el9 ceph 175 k 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction Summary 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:================================================================================ 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Install 1 Package 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Total download size: 175 k 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Installed size: 673 k 2024-04-16T06:01:44.662 INFO:teuthology.orchestra.run.smithi169.stdout:Downloading Packages: 2024-04-16T06:01:44.880 INFO:teuthology.orchestra.run.smithi169.stdout:rbd-nbd-19.0.0-3085.gc0a28781.el9.x86_64.rpm 805 kB/s | 175 kB 00:00 2024-04-16T06:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout:-------------------------------------------------------------------------------- 2024-04-16T06:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout:Total 800 kB/s | 175 kB 00:00 2024-04-16T06:01:44.881 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction check 2024-04-16T06:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction check succeeded. 2024-04-16T06:01:44.891 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction test 2024-04-16T06:01:44.947 INFO:teuthology.orchestra.run.smithi169.stdout:Transaction test succeeded. 2024-04-16T06:01:44.948 INFO:teuthology.orchestra.run.smithi169.stdout:Running transaction 2024-04-16T06:01:45.094 INFO:teuthology.orchestra.run.smithi169.stdout: Preparing : 1/1 2024-04-16T06:01:45.204 INFO:teuthology.orchestra.run.smithi169.stdout: Installing : rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.642 INFO:teuthology.orchestra.run.smithi169.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout: Verifying : rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 1/1 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout:Installed: 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout: rbd-nbd-2:19.0.0-3085.gc0a28781.el9.x86_64 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:45.936 INFO:teuthology.orchestra.run.smithi169.stdout:Complete! 2024-04-16T06:01:46.198 DEBUG:teuthology.parallel:result is None 2024-04-16T06:01:46.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:01:46.340 DEBUG:teuthology.orchestra.run.smithi169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-16T06:01:46.362 INFO:teuthology.orchestra.run.smithi169.stdout:19.0.0-3085.gc0a28781.el9 2024-04-16T06:01:46.363 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3085.gc0a28781.el9 2024-04-16T06:01:46.363 INFO:teuthology.task.install:The correct ceph version 19.0.0-3085.gc0a28781 is installed. 2024-04-16T06:01:46.364 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-16T06:01:46.364 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:46.365 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-16T06:01:46.431 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-16T06:01:46.431 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:46.431 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/daemon-helper 2024-04-16T06:01:46.499 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-16T06:01:46.567 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-16T06:01:46.568 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:46.568 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-16T06:01:46.636 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-16T06:01:46.704 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-16T06:01:46.704 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:46.704 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/usr/bin/stdin-killer 2024-04-16T06:01:46.771 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-16T06:01:46.840 INFO:teuthology.run_tasks:Running task ceph... 2024-04-16T06:01:46.922 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-16T06:01:46.922 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 777 /var/log/ceph 2024-04-16T06:01:46.950 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-16T06:01:46.950 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-16T06:01:47.019 INFO:tasks.ceph:Creating extra log directories... 2024-04-16T06:01:47.019 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-16T06:01:47.094 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-16T06:01:47.094 INFO:tasks.ceph:config {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-16T06:01:47.094 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937', 'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'description': 'crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format}', 'email': None, 'first_in_suite': False, 'job_id': '7657937', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'}}, 'owner': 'scheduled_nmordech@teuthology', 'priority': 50, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 9883, 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'sleep_before_teardown': 0, 'suite': 'crimson-rados:rbd', 'suite_branch': 'wip-nitzan-seastore-re-enable-gate-debug-flag', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'targets': {'smithi169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaUGh9tzksP6/9SThjKLXdcAk18nfXCKPCFuUruX739LRCFwz928P9bstiBxnORvfJXtxjYL/l+Hyb1IrYXIVs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'mon': {'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'osd pool default crimson': True}, 'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}}, 'flavor': 'crimson', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'is full \\(reached quota', '\\(POOL_FULL\\)'], 'sha1': 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc', 'cluster': 'ceph'}}, {'workunit': {'clients': {'client.0': ['rbd/crimson/test_crimson_librbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-16_05:45:04', 'tube': 'smithi', 'user': 'nmordech', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-16T06:01:47.095 DEBUG:teuthology.orchestra.run.smithi169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-16T06:01:47.143 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -d -m0777 -- /var/run/ceph 2024-04-16T06:01:47.212 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:47.213 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/scratch_devs of=/dev/stdout 2024-04-16T06:01:47.269 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-16T06:01:47.270 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_1 2024-04-16T06:01:47.325 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-16 06:01:42.671580348 +0000 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-16 05:58:08.453861609 +0000 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-16 05:58:08.453861609 +0000 2024-04-16T06:01:47.326 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-16 05:58:08.453861609 +0000 2024-04-16T06:01:47.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-16T06:01:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-16T06:01:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-16T06:01:47.392 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000198074 s, 2.6 MB/s 2024-04-16T06:01:47.393 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-16T06:01:47.450 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_2 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-16 06:01:42.672580328 +0000 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-16 05:58:08.803854354 +0000 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-16 05:58:08.803854354 +0000 2024-04-16T06:01:47.505 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-16 05:58:08.803854354 +0000 2024-04-16T06:01:47.506 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-16T06:01:47.571 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-16T06:01:47.571 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-16T06:01:47.571 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.00021235 s, 2.4 MB/s 2024-04-16T06:01:47.572 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-16T06:01:47.629 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_3 2024-04-16T06:01:47.685 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-16T06:01:47.685 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:01:47.685 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-16T06:01:47.685 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:01:47.685 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:01:47.686 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-16 06:01:42.672580328 +0000 2024-04-16T06:01:47.686 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-16 05:58:09.141847348 +0000 2024-04-16T06:01:47.686 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-16 05:58:09.141847348 +0000 2024-04-16T06:01:47.686 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-16 05:58:09.141847348 +0000 2024-04-16T06:01:47.686 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-16T06:01:47.752 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-16T06:01:47.753 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-16T06:01:47.753 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000218379 s, 2.3 MB/s 2024-04-16T06:01:47.754 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-16T06:01:47.810 DEBUG:teuthology.orchestra.run.smithi169:> stat /dev/vg_nvme/lv_4 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Context: system_u:object_r:device_t:s0 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Access: 2024-04-16 06:01:42.672580328 +0000 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Modify: 2024-04-16 05:58:09.482840280 +0000 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout:Change: 2024-04-16 05:58:09.482840280 +0000 2024-04-16T06:01:47.865 INFO:teuthology.orchestra.run.smithi169.stdout: Birth: 2024-04-16 05:58:09.482840280 +0000 2024-04-16T06:01:47.865 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-16T06:01:47.931 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records in 2024-04-16T06:01:47.931 INFO:teuthology.orchestra.run.smithi169.stderr:1+0 records out 2024-04-16T06:01:47.931 INFO:teuthology.orchestra.run.smithi169.stderr:512 bytes copied, 0.000188286 s, 2.7 MB/s 2024-04-16T06:01:47.932 DEBUG:teuthology.orchestra.run.smithi169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-16T06:01:47.989 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:01:47.989 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T06:01:47.989 INFO:tasks.ceph:Generating config... 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-16T06:01:47.990 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-16T06:01:47.991 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-16T06:01:47.991 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-16T06:01:47.991 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-16T06:01:47.991 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-16T06:01:47.992 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-16T06:01:47.992 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-16T06:01:47.992 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-1 2024-04-16T06:01:47.992 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 2-3 2024-04-16T06:01:47.992 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 4-5 2024-04-16T06:01:47.992 INFO:tasks.ceph:Setting up mon.a... 2024-04-16T06:01:47.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-16T06:01:48.111 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.keyring 2024-04-16T06:01:48.114 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-16T06:01:48.168 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T06:01:48.195 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.169')] 2024-04-16T06:01:48.196 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.169', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10, 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-1'}, 'osd.1': {'crimson seastar cpu cores': '2-3'}, 'osd.2': {'crimson seastar cpu cores': '4-5'}} 2024-04-16T06:01:48.196 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:48.196 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T06:01:48.251 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.169 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:01:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:01:48.343+0000 7fc9c2859f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:01:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:01:48.343+0000 7fc9c2859f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:01:48.342 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:01:48.343+0000 7fc9c2859f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:01:48.342 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:01:48.342 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: generated fsid 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:setting min_mon_release = quincy 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:epoch 0 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:fsid 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:last_changed 2024-04-16T06:01:48.345133+0000 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:created 2024-04-16T06:01:48.345133+0000 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:min_mon_release 17 (quincy) 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:election_strategy: 1 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:0: [v2:172.21.15.169:3300/0,v1:172.21.15.169:6789/0] mon.a 2024-04-16T06:01:48.343 INFO:teuthology.orchestra.run.smithi169.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-16T06:01:48.344 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-16T06:01:48.359 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 273ea366-5e3c-48b2-b98b-be01021c58a8... 2024-04-16T06:01:48.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout:[global] 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: chdir = "" 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: auth supported = cephx 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: filestore xattr use omap = true 2024-04-16T06:01:48.453 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: mon clock drift allowed = 1.000 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: osd crush chooseleaf type = 0 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: auth debug = true 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on old message = true 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: ms die on bug = true 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: mon pg warn max object skew = 0 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: # disable pg_autoscaler by default for new pools 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.454 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default size = 2 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow primary affinity = true 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd allow pg remap = true 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on legacy crush tunables = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on crush straw calc version zero = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on no sortbitwise = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on osd down out interval zero = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon warn on too few osds = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: mon_allow_pool_size_one = true 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: osd default data pool replay window = 5 2024-04-16T06:01:48.455 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon allow pool delete = true 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon cluster log file level = debug 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: debug asserts on shutdown = true 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon health detail to clog = false 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: mon host = 172.21.15.169 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms client mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms cluster mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms mon client mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms mon cluster mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms mon service mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: ms service mode = crc 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: fsid = 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.456 INFO:teuthology.orchestra.run.smithi169.stdout:[osd] 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd journal size = 100 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub load threshold = 5.0 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd scrub max interval = 600 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd mclock profile = high_recovery_ops 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd recover clone overlap = true 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd recovery max chunk = 1048576 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug shutdown = true 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug op order = true 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug verify stray on activate = true 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug trim objects = true 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.457 INFO:teuthology.orchestra.run.smithi169.stdout: osd open classes on start = true 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug pg log writeout = true 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: osd deep scrub update digest min age = 30 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: osd map max advance = 10 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: journal zero on create = true 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: filestore ondisk finisher threads = 3 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: filestore apply finisher threads = 3 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: bdev debug aio = true 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: osd debug misdirected ops = true 2024-04-16T06:01:48.458 INFO:teuthology.orchestra.run.smithi169.stdout: debug monc = 20 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: crimson alien thread cpu cores = 6-7 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: crimson osd obc lru size = 10 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug osd = 20 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: osd shutdown pgref assert = True 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr] 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug mgr = 20 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: mgr/telemetry/nag = false 2024-04-16T06:01:48.459 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout:[mon] 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: debug ms = 1 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: debug mon = 20 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: debug paxos = 20 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: debug auth = 20 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon data avail warn = 5 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon mgr mkfs grace = 240 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min pgs per osd = 4 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd reporter subtree level = osd 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon osd prime pg temp = true 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: mon reweight min bytes per osd = 10 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: auth mon ticket ttl = 660 # 11m 2024-04-16T06:01:48.460 INFO:teuthology.orchestra.run.smithi169.stdout: auth service ticket ttl = 240 # 4m 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: # don't complain about insecure global_id in the test suite 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: # 1m isn't quite enough 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon_down_mkfs_grace = 2m 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon_warn_on_filestore_osds = false 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon min osdmap epochs = 50 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune interval = 2 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune min = 15 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: mon osdmap full prune txsize = 2 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: paxos service trim min = 10 2024-04-16T06:01:48.461 INFO:teuthology.orchestra.run.smithi169.stdout: osd pool default crimson = True 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout:[client] 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: rgw cache enabled = true 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable ops log = true 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: rgw enable usage log = true 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout:[mon.a] 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: crimson seastar cpu cores = 0-1 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: crimson seastar cpu cores = 2-3 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-04-16T06:01:48.462 INFO:teuthology.orchestra.run.smithi169.stdout: crimson seastar cpu cores = 4-5 2024-04-16T06:01:48.468 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-16T06:01:48.468 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-16T06:01:48.576 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-16T06:01:48.576 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:48.576 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-16T06:01:48.592 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:48.592 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-16T06:01:48.647 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T06:01:48.647 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:48.647 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-16T06:01:48.648 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-16T06:01:48.729 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:01:48.729 DEBUG:teuthology.orchestra.run.smithi169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:01:48.785 INFO:tasks.ceph:Setting up mon nodes... 2024-04-16T06:01:48.785 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-16T06:01:48.785 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T06:01:48.891 INFO:teuthology.orchestra.run.smithi169.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-16T06:01:48.894 INFO:tasks.ceph:Setting up mds nodes... 2024-04-16T06:01:48.894 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-16T06:01:48.894 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-16T06:01:48.947 INFO:teuthology.orchestra.run.smithi169.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-16T06:01:48.964 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-16T06:01:48.964 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-16T06:01:48.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-16T06:01:48.991 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:01:48.991 INFO:tasks.ceph:role: osd.0 2024-04-16T06:01:48.991 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T06:01:48.991 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:01:49.067 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:01:49.573 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-04-16T06:01:49.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-04-16T06:01:49.575 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-16T06:01:49.731 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-16T06:01:49.769 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-16T06:01:49.840 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:01:49.840 INFO:tasks.ceph:role: osd.1 2024-04-16T06:01:49.840 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T06:01:49.840 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:01:49.917 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:01:50.363 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-04-16T06:01:50.366 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-04-16T06:01:50.366 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-16T06:01:50.462 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-16T06:01:50.493 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-16T06:01:50.562 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-16T06:01:50.562 INFO:tasks.ceph:role: osd.2 2024-04-16T06:01:50.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T06:01:50.562 DEBUG:teuthology.orchestra.run.smithi169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-16T06:01:50.637 INFO:teuthology.orchestra.run.smithi169.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-16T06:01:50.637 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-16T06:01:50.637 INFO:teuthology.orchestra.run.smithi169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-16T06:01:50.637 INFO:teuthology.orchestra.run.smithi169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-16T06:01:50.637 INFO:teuthology.orchestra.run.smithi169.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-16T06:01:50.638 INFO:teuthology.orchestra.run.smithi169.stdout: = sunit=0 swidth=0 blks 2024-04-16T06:01:50.638 INFO:teuthology.orchestra.run.smithi169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-16T06:01:50.638 INFO:teuthology.orchestra.run.smithi169.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-16T06:01:50.638 INFO:teuthology.orchestra.run.smithi169.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-16T06:01:50.638 INFO:teuthology.orchestra.run.smithi169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-16T06:01:51.137 INFO:teuthology.orchestra.run.smithi169.stdout:Discarding blocks...Done. 2024-04-16T06:01:51.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi169.front.sepia.ceph.com -o noatime 2024-04-16T06:01:51.140 DEBUG:teuthology.orchestra.run.smithi169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-16T06:01:51.291 DEBUG:teuthology.orchestra.run.smithi169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-16T06:01:51.326 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:01:51.642 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:01:51.644 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:51,647 seastar - Reactor backend: linux-aio 2024-04-16T06:01:51.655 INFO:teuthology.orchestra.run.smithi169.stderr:==29218==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:01:51.773 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:51,776 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:01:51.773 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:51,776 [shard 0:main] osd - get_early_config: set --cpuset 0-1 2024-04-16T06:01:51.773 INFO:teuthology.orchestra.run.smithi169.stderr:DEBUG 2024-04-16 06:01:51,777 [shard 0:main] seastar - reactor::drain 2024-04-16T06:01:52.333 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:52.333 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:01:52.333 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:01:52.334 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:01:52.334 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:52.334 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:01:52.389 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:01:52.393 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:52,396 seastar - Reactor backend: linux-aio 2024-04-16T06:01:52.401 INFO:teuthology.orchestra.run.smithi169.stderr:==29216==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:01:55.846 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:01:55.847 INFO:teuthology.orchestra.run.smithi169.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:01:55.873 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-16T06:01:55.901 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:01:56.202 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:01:56.203 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:56,205 seastar - Reactor backend: linux-aio 2024-04-16T06:01:56.210 INFO:teuthology.orchestra.run.smithi169.stderr:==30024==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:01:56.334 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:56,337 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:01:56.334 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:56,337 [shard 0:main] osd - get_early_config: set --cpuset 2-3 2024-04-16T06:01:56.334 INFO:teuthology.orchestra.run.smithi169.stderr:DEBUG 2024-04-16 06:01:56,338 [shard 0:main] seastar - reactor::drain 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:01:56.870 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:01:56.926 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:01:56.930 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:01:56,933 seastar - Reactor backend: linux-aio 2024-04-16T06:01:56.938 INFO:teuthology.orchestra.run.smithi169.stderr:==30022==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:02:00.389 INFO:teuthology.orchestra.run.smithi169.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:02:00.413 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-16T06:02:00.441 DEBUG:teuthology.orchestra.run.smithi169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:00.751 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:00.752 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:02:00,755 seastar - Reactor backend: linux-aio 2024-04-16T06:02:00.757 INFO:teuthology.orchestra.run.smithi169.stderr:==30830==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:00.864 INFO:teuthology.orchestra.run.smithi169.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb9a202b 0xb874970 0xb875eec 0xb876102 0xb876258 0xb876721 0x54daf 0x368e4 0x3fc40a7 0x5d87d8e 0x5d87f8a 0x5d88516 0x5bccfdf 0x5be80a3 0x6ec2944 0x6ed4117 0x6ed55b4 0x6ed5639 0x6ed573b 0x6ed5833 0x6ed5a40 0x6ed5c26 0x6ed5f75 0x6ed6319 0x6ed6549 0x6ed67b3 0x6ed6a80 0x6ed6c9e 0x6ed6d56 0x6ed6e1c 0x6ed6efd 0x5a358de 0x5a359e8 0x5a35aa4 0x5a35b60 0x5a35c1c 0x5a35d69 0x5a35ef4 0x5a362b5 0x5a521f6 0xb8876e3 0xb8a197d 0xb942a0f 0xb94405d 0xb6206c1 0xb62103d 0x37d7bfc 0x37d8755 0x3678150 0x3feaf 0x3ff5f 0x3462444 2024-04-16T06:02:00.864 INFO:teuthology.orchestra.run.smithi169.stderr:kernel callstack: 2024-04-16T06:02:00.880 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:02:00,882 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2024-04-16T06:02:00.880 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:02:00,883 [shard 0:main] osd - get_early_config: set --cpuset 4-5 2024-04-16T06:02:00.880 INFO:teuthology.orchestra.run.smithi169.stderr:DEBUG 2024-04-16 06:02:00,883 [shard 0:main] seastar - reactor::drain 2024-04-16T06:02:01.427 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:01.428 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:02:01.428 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:02:01.428 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:01.428 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:01.428 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:02:01.490 INFO:teuthology.orchestra.run.smithi169.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-16T06:02:01.493 INFO:teuthology.orchestra.run.smithi169.stderr:INFO 2024-04-16 06:02:01,496 seastar - Reactor backend: linux-aio 2024-04-16T06:02:01.500 INFO:teuthology.orchestra.run.smithi169.stderr:==30828==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr:Suppressions used: 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr: count bytes template 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr: 1 8 MallocExtension::Register 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr:----------------------------------------------------- 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T06:02:04.939 INFO:teuthology.orchestra.run.smithi169.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 273ea366-5e3c-48b2-b98b-be01021c58a8 2024-04-16T06:02:04.964 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-16T06:02:04.992 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-16T06:02:04.992 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:04.992 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-16T06:02:05.061 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:05.061 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-16T06:02:05.129 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:05.129 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-16T06:02:05.197 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:05.197 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-16T06:02:05.265 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:05.265 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-16T06:02:05.320 INFO:tasks.ceph:Adding keys to all mons... 2024-04-16T06:02:05.320 DEBUG:teuthology.orchestra.run.smithi169:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout:[mgr.x] 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBMFB5mHe1RNRAARfxaQF2QS6AezdwtPkf1Ww== 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.0] 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBQFB5mra6AIBAA65V10nOWBiGPdsgyprBdwA== 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.1] 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBVFB5m4GPYBBAALaKQdnxPqzyW+8g4ry3GmA== 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout:[osd.2] 2024-04-16T06:02:05.386 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBZFB5mpZ2VJhAA1hEYXsQlrpP3iyfU7RfCQA== 2024-04-16T06:02:05.387 INFO:teuthology.orchestra.run.smithi169.stdout:[client.0] 2024-04-16T06:02:05.387 INFO:teuthology.orchestra.run.smithi169.stdout: key = AQBMFB5m0simOBAA/fA9AAEUjDmrhTMHnm3HIQ== 2024-04-16T06:02:05.387 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-16T06:02:05.491 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:05.555 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:05.619 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-16T06:02:05.681 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-16T06:02:05.748 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-16T06:02:05.748 DEBUG:teuthology.orchestra.run.smithi169:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-16T06:02:05.776 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-16T06:02:05.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:05.894+0000 7f1e23271a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:05.892 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:05.894+0000 7f1e23271a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:05.909 DEBUG:teuthology.orchestra.run.smithi169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-16T06:02:05.937 DEBUG:teuthology.orchestra.run.smithi169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-16T06:02:05.993 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-16T06:02:05.993 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-16T06:02:05.993 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-16T06:02:06.035 INFO:tasks.ceph.mon.a:Started 2024-04-16T06:02:06.035 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-16T06:02:06.035 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-16T06:02:06.036 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-16T06:02:06.038 INFO:tasks.ceph.mgr.x:Started 2024-04-16T06:02:06.038 INFO:tasks.ceph:Setting crush tunables to default 2024-04-16T06:02:06.038 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd crush tunables default 2024-04-16T06:02:06.122 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:02:06.124+0000 7fdd9405fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.122 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:02:06.124+0000 7fdd9405fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.122 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:02:06.124+0000 7fdd9405fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.123 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:06.125+0000 7f6e7a57f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.154 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:06.151+0000 7fce1cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.159 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:06.161+0000 7fce1cf9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.230 INFO:teuthology.orchestra.run.smithi169.stderr:adjusted tunables profile to default 2024-04-16T06:02:06.241 INFO:tasks.ceph:check_enable_crimson: True 2024-04-16T06:02:06.242 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-16T06:02:06.242 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-16T06:02:06.328 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:06.330+0000 7f6e7a57f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.328 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:06.330+0000 7f6e7a57f000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.353 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:06.350+0000 7fdeaac06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.356 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:06.358+0000 7fdeaac06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:02:06.439+0000 7fdd8adf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr:Please be aware that this feature is experimental, untested, 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr:unsupported, and may result in data corruption, data loss, 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr:feature with important data. 2024-04-16T06:02:06.438 INFO:tasks.ceph.mon.a.smithi169.stderr: 2024-04-16T06:02:06.857 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:06.859+0000 7f6e7a57f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-16T06:02:06.911 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:06.913+0000 7f6e7a57f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-16T06:02:07.243 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-16T06:02:07.243 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:07.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-16T06:02:07.272 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:07.272 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-16T06:02:07.339 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:02:07.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-16T06:02:07.408 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new feba7000-73bd-4caf-8585-9a7d5721fa4e 0 2024-04-16T06:02:07.498 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.500+0000 7f6e7a57f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-16T06:02:07.558 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.560+0000 7f6e7a57f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-16T06:02:07.563 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:07.558+0000 7fdfe666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:07.567 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:07.569+0000 7fdfe666d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:07.607 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.609+0000 7f6e7a57f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-16T06:02:07.652 INFO:teuthology.orchestra.run.smithi169.stdout:0 2024-04-16T06:02:07.664 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 99a32654-fdc1-49a2-8648-2f79c1787261 1 2024-04-16T06:02:07.666 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.667+0000 7f6e7a57f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-16T06:02:07.713 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.715+0000 7f6e7a57f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-16T06:02:07.779 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:07.776+0000 7fe4474ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:07.784 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:07.786+0000 7fe4474ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:07.842 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.843+0000 7f6e7a57f000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-16T06:02:07.868 INFO:teuthology.orchestra.run.smithi169.stdout:1 2024-04-16T06:02:07.880 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd new 62efc5b4-8bfd-444b-8848-f6db43af5218 2 2024-04-16T06:02:07.918 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.920+0000 7f6e7a57f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-16T06:02:07.969 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:07.971+0000 7f6e7a57f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-16T06:02:07.994 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:07.990+0000 7fda4a624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:07.998 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:08.000+0000 7fda4a624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:08.026 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.028+0000 7f6e7a57f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-16T06:02:08.075 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.077+0000 7f6e7a57f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-16T06:02:08.081 INFO:teuthology.orchestra.run.smithi169.stdout:2 2024-04-16T06:02:08.093 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-16T06:02:08.167 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-16T06:02:08.167 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-16T06:02:08.209 INFO:tasks.ceph.osd.0:Started 2024-04-16T06:02:08.209 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-16T06:02:08.243 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-16T06:02:08.243 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-16T06:02:08.287 INFO:tasks.ceph.osd.1:Started 2024-04-16T06:02:08.288 DEBUG:teuthology.orchestra.run.smithi169:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-16T06:02:08.337 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-16T06:02:08.337 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-16T06:02:08.340 INFO:tasks.ceph.osd.2:Started 2024-04-16T06:02:08.340 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:08.468 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:08.465+0000 7f88bde20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:08.472 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:08.475+0000 7f88bde20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:08.568 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.569+0000 7f6e7a57f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-16T06:02:08.606 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:08.606 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":6,"fsid":"273ea366-5e3c-48b2-b98b-be01021c58a8","created":"2024-04-16T06:02:06.148853+0000","modified":"2024-04-16T06:02:08.083426+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T06:02:08.083426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"feba7000-73bd-4caf-8585-9a7d5721fa4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99a32654-fdc1-49a2-8648-2f79c1787261","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62efc5b4-8bfd-444b-8848-f6db43af5218","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:08.628 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-16T06:02:08.629 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-16T06:02:08.725 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.727+0000 7f6e7a57f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-16T06:02:08.844 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.846+0000 7f6e7a57f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-16T06:02:08.893 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:08.895+0000 7f6e7a57f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-16T06:02:08.931 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:09.018 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:09.020+0000 7f6e7a57f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-16T06:02:09.036 INFO:teuthology.misc.health.smithi169.stderr:2024-04-16T06:02:09.034+0000 7f99ac55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:09.040 INFO:teuthology.misc.health.smithi169.stderr:2024-04-16T06:02:09.042+0000 7f99ac55d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:09.123 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:09.125+0000 7f6e7a57f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-16T06:02:09.169 INFO:teuthology.misc.health.smithi169.stdout: 2024-04-16T06:02:09.169 INFO:teuthology.misc.health.smithi169.stdout:{"epoch":6,"fsid":"273ea366-5e3c-48b2-b98b-be01021c58a8","created":"2024-04-16T06:02:06.148853+0000","modified":"2024-04-16T06:02:08.083426+0000","last_up_change":"0.000000","last_in_change":"2024-04-16T06:02:08.083426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"feba7000-73bd-4caf-8585-9a7d5721fa4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"99a32654-fdc1-49a2-8648-2f79c1787261","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"62efc5b4-8bfd-444b-8848-f6db43af5218","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:09.183 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-16T06:02:09.295 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:09.297+0000 7f6e7a57f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-16T06:02:09.343 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:09.345+0000 7f6e7a57f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-16T06:02:09.515 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:09.517+0000 7f6e7a57f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-16T06:02:10.620 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:10.622+0000 7f6e7a57f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-16T06:02:10.925 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:10.927+0000 7f6e7a57f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-16T06:02:10.989 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:10.991+0000 7f6e7a57f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-16T06:02:11.130 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:11.131+0000 7f6e7a57f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-16T06:02:12.139 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:02:12.140+0000 7f6e197fc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-16T06:02:15.485 DEBUG:teuthology.orchestra.run.smithi169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:15.594 INFO:teuthology.misc.health.smithi169.stderr:2024-04-16T06:02:15.592+0000 7f270939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:15.598 INFO:teuthology.misc.health.smithi169.stderr:2024-04-16T06:02:15.600+0000 7f270939d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:15.786 INFO:teuthology.misc.health.smithi169.stdout: 2024-04-16T06:02:15.786 INFO:teuthology.misc.health.smithi169.stdout:{"epoch":9,"fsid":"273ea366-5e3c-48b2-b98b-be01021c58a8","created":"2024-04-16T06:02:06.148853+0000","modified":"2024-04-16T06:02:15.201766+0000","last_up_change":"2024-04-16T06:02:13.194890+0000","last_in_change":"2024-04-16T06:02:08.083426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:15.158551+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feba7000-73bd-4caf-8585-9a7d5721fa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":1397559365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1397559365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1397559365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1397559365}]},"public_addr":"172.21.15.169:6805/1397559365","cluster_addr":"172.21.15.169:6804/1397559365","heartbeat_back_addr":"172.21.15.169:6814/1397559365","heartbeat_front_addr":"172.21.15.169:6813/1397559365","state":["exists","up"]},{"osd":1,"uuid":"99a32654-fdc1-49a2-8648-2f79c1787261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":2019538248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":2019538248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2019538248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2019538248}]},"public_addr":"172.21.15.169:6803/2019538248","cluster_addr":"172.21.15.169:6802/2019538248","heartbeat_back_addr":"172.21.15.169:6810/2019538248","heartbeat_front_addr":"172.21.15.169:6811/2019538248","state":["exists","up"]},{"osd":2,"uuid":"62efc5b4-8bfd-444b-8848-f6db43af5218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":1013263003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1013263003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1013263003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1013263003}]},"public_addr":"172.21.15.169:6807/1013263003","cluster_addr":"172.21.15.169:6808/1013263003","heartbeat_back_addr":"172.21.15.169:6812/1013263003","heartbeat_front_addr":"172.21.15.169:6809/1013263003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:15.798 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-16T06:02:15.798 INFO:tasks.ceph:Creating RBD pool 2024-04-16T06:02:15.798 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-16T06:02:15.912 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:15.907+0000 7f9c64f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:15.915 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:15.917+0000 7f9c64f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:16.203 INFO:teuthology.orchestra.run.smithi169.stderr:pool 'rbd' created 2024-04-16T06:02:16.224 DEBUG:teuthology.orchestra.run.smithi169:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-16T06:02:16.364 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:16.356+0000 7f68e4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:16.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:16.371+0000 7f68e4d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:17.233 INFO:teuthology.orchestra.run.smithi169.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-16T06:02:17.253 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-16T06:02:17.253 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-16T06:02:17.253 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-16T06:02:17.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-16T06:02:17.254 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-16T06:02:17.407 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:17.405+0000 7f781099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:17.411 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:17.414+0000 7f781099d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:17.666 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:17.682 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6800","nonce":2611156474},{"type":"v1","addr":"172.21.15.169:6801","nonce":2611156474}]},"active_addr":"172.21.15.169:6801/2611156474","active_change":"2024-04-16T06:02:11.135101+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":2868146760}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":2972748481}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":2404984510}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.169:0","nonce":2118129496}]}]} 2024-04-16T06:02:17.684 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-16T06:02:17.684 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-16T06:02:17.684 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:17.801 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:17.800+0000 7fa9fee64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:17.805 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:17.807+0000 7fa9fee64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:17.996 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:17.996 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":11,"fsid":"273ea366-5e3c-48b2-b98b-be01021c58a8","created":"2024-04-16T06:02:06.148853+0000","modified":"2024-04-16T06:02:17.227998+0000","last_up_change":"2024-04-16T06:02:13.194890+0000","last_in_change":"2024-04-16T06:02:08.083426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:15.158551+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T06:02:16.108948+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feba7000-73bd-4caf-8585-9a7d5721fa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":1397559365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1397559365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1397559365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1397559365}]},"public_addr":"172.21.15.169:6805/1397559365","cluster_addr":"172.21.15.169:6804/1397559365","heartbeat_back_addr":"172.21.15.169:6814/1397559365","heartbeat_front_addr":"172.21.15.169:6813/1397559365","state":["exists","up"]},{"osd":1,"uuid":"99a32654-fdc1-49a2-8648-2f79c1787261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":2019538248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":2019538248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2019538248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2019538248}]},"public_addr":"172.21.15.169:6803/2019538248","cluster_addr":"172.21.15.169:6802/2019538248","heartbeat_back_addr":"172.21.15.169:6810/2019538248","heartbeat_front_addr":"172.21.15.169:6811/2019538248","state":["exists","up"]},{"osd":2,"uuid":"62efc5b4-8bfd-444b-8848-f6db43af5218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":1013263003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1013263003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1013263003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1013263003}]},"public_addr":"172.21.15.169:6807/1013263003","cluster_addr":"172.21.15.169:6808/1013263003","heartbeat_back_addr":"172.21.15.169:6812/1013263003","heartbeat_front_addr":"172.21.15.169:6809/1013263003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:18.010 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-16T06:02:18.011 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-16T06:02:18.123 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.122+0000 7ff78e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.127 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.129+0000 7ff78e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.325 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:18.325 INFO:teuthology.orchestra.run.smithi169.stdout:{"epoch":12,"fsid":"273ea366-5e3c-48b2-b98b-be01021c58a8","created":"2024-04-16T06:02:06.148853+0000","modified":"2024-04-16T06:02:18.236151+0000","last_up_change":"2024-04-16T06:02:13.194890+0000","last_in_change":"2024-04-16T06:02:08.083426+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-16T06:02:15.158551+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-16T06:02:16.108948+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"feba7000-73bd-4caf-8585-9a7d5721fa4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6805","nonce":1397559365}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6804","nonce":1397559365}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6814","nonce":1397559365}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6813","nonce":1397559365}]},"public_addr":"172.21.15.169:6805/1397559365","cluster_addr":"172.21.15.169:6804/1397559365","heartbeat_back_addr":"172.21.15.169:6814/1397559365","heartbeat_front_addr":"172.21.15.169:6813/1397559365","state":["exists","up"]},{"osd":1,"uuid":"99a32654-fdc1-49a2-8648-2f79c1787261","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6803","nonce":2019538248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6802","nonce":2019538248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6810","nonce":2019538248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6811","nonce":2019538248}]},"public_addr":"172.21.15.169:6803/2019538248","cluster_addr":"172.21.15.169:6802/2019538248","heartbeat_back_addr":"172.21.15.169:6810/2019538248","heartbeat_front_addr":"172.21.15.169:6811/2019538248","state":["exists","up"]},{"osd":2,"uuid":"62efc5b4-8bfd-444b-8848-f6db43af5218","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6807","nonce":1013263003}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6808","nonce":1013263003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6812","nonce":1013263003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.169:6809","nonce":1013263003}]},"public_addr":"172.21.15.169:6807/1013263003","cluster_addr":"172.21.15.169:6808/1013263003","heartbeat_back_addr":"172.21.15.169:6812/1013263003","heartbeat_front_addr":"172.21.15.169:6809/1013263003","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-16T06:02:18.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-16T06:02:18.338 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-16T06:02:18.339 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-16T06:02:18.458 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.456+0000 7f0e1499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.456+0000 7fcc0626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.459 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.456+0000 7f59444ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.465+0000 7fcc0626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.465+0000 7f0e1499d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.463 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.465+0000 7f59444ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.497 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.498 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-16T06:02:18.501 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.501 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-16T06:02:18.503 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.503 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-16T06:02:18.614 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.612+0000 7f253f824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.619 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.621+0000 7f253f824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.659+0000 7fed43a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.662 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.660+0000 7f3a2339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.669+0000 7f3a2339d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.667 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:18.669+0000 7fed43a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:18.821 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.834 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-04-16T06:02:18.834 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:18.898 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.906 INFO:teuthology.orchestra.run.smithi169.stdout:34359738373 2024-04-16T06:02:18.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-04-16T06:02:18.913 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:18.919 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-04-16T06:02:18.919 DEBUG:teuthology.parallel:result is None 2024-04-16T06:02:18.919 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T06:02:18.920 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T06:02:19.073 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.071+0000 7f544c99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.076 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.078+0000 7f544c99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.262 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:19.263 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-04-16T06:02:19.275 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T06:02:19.140028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82668,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490132,"statfs":{"total":322122547200,"available":322037895168,"internally_reserved":0,"allocated":1470464,"data_stored":1245350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.652488"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.355005+0000","last_change":"2024-04-16T06:02:17.356153+0000","last_active":"2024-04-16T06:02:17.355005+0000","last_peered":"2024-04-16T06:02:17.355005+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.349456+0000","last_became_peered":"2024-04-16T06:02:17.349456+0000","last_unstale":"2024-04-16T06:02:17.355005+0000","last_undegraded":"2024-04-16T06:02:17.355005+0000","last_fullsized":"2024-04-16T06:02:17.355005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.336857+0000","last_change":"2024-04-16T06:02:17.338547+0000","last_active":"2024-04-16T06:02:17.336857+0000","last_peered":"2024-04-16T06:02:17.336857+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.333212+0000","last_became_peered":"2024-04-16T06:02:17.333212+0000","last_unstale":"2024-04-16T06:02:17.336857+0000","last_undegraded":"2024-04-16T06:02:17.336857+0000","last_fullsized":"2024-04-16T06:02:17.336857+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.331530+0000","last_change":"2024-04-16T06:02:17.332764+0000","last_active":"2024-04-16T06:02:17.331530+0000","last_peered":"2024-04-16T06:02:17.331530+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.325662+0000","last_became_peered":"2024-04-16T06:02:17.325662+0000","last_unstale":"2024-04-16T06:02:17.331530+0000","last_undegraded":"2024-04-16T06:02:17.331530+0000","last_fullsized":"2024-04-16T06:02:17.331530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.316521+0000","last_change":"2024-04-16T06:02:17.317578+0000","last_active":"2024-04-16T06:02:17.316521+0000","last_peered":"2024-04-16T06:02:17.316521+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.308308+0000","last_became_peered":"2024-04-16T06:02:17.308308+0000","last_unstale":"2024-04-16T06:02:17.316521+0000","last_undegraded":"2024-04-16T06:02:17.316521+0000","last_fullsized":"2024-04-16T06:02:17.316521+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.332054+0000","last_change":"2024-04-16T06:02:17.333634+0000","last_active":"2024-04-16T06:02:17.332054+0000","last_peered":"2024-04-16T06:02:17.332054+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.327541+0000","last_became_peered":"2024-04-16T06:02:17.327541+0000","last_unstale":"2024-04-16T06:02:17.332054+0000","last_undegraded":"2024-04-16T06:02:17.332054+0000","last_fullsized":"2024-04-16T06:02:17.332054+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:18.256234+0000","last_change":"2024-04-16T06:02:17.344127+0000","last_active":"2024-04-16T06:02:18.256234+0000","last_peered":"2024-04-16T06:02:18.256234+0000","last_clean":"2024-04-16T06:02:18.256234+0000","last_became_active":"2024-04-16T06:02:17.339028+0000","last_became_peered":"2024-04-16T06:02:17.339028+0000","last_unstale":"2024-04-16T06:02:18.256234+0000","last_undegraded":"2024-04-16T06:02:18.256234+0000","last_fullsized":"2024-04-16T06:02:18.256234+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:18.253126+0000","last_change":"2024-04-16T06:02:17.325482+0000","last_active":"2024-04-16T06:02:18.253126+0000","last_peered":"2024-04-16T06:02:18.253126+0000","last_clean":"2024-04-16T06:02:18.253126+0000","last_became_active":"2024-04-16T06:02:17.319053+0000","last_became_peered":"2024-04-16T06:02:17.319053+0000","last_unstale":"2024-04-16T06:02:18.253126+0000","last_undegraded":"2024-04-16T06:02:18.253126+0000","last_fullsized":"2024-04-16T06:02:18.253126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.320870+0000","last_change":"2024-04-16T06:02:17.322229+0000","last_active":"2024-04-16T06:02:17.320870+0000","last_peered":"2024-04-16T06:02:17.320870+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.317282+0000","last_became_peered":"2024-04-16T06:02:17.317282+0000","last_unstale":"2024-04-16T06:02:17.320870+0000","last_undegraded":"2024-04-16T06:02:17.320870+0000","last_fullsized":"2024-04-16T06:02:17.320870+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.719717+0000","last_change":"2024-04-16T06:02:16.257389+0000","last_active":"2024-04-16T06:02:17.719717+0000","last_peered":"2024-04-16T06:02:17.719717+0000","last_clean":"2024-04-16T06:02:17.719717+0000","last_became_active":"2024-04-16T06:02:16.252428+0000","last_became_peered":"2024-04-16T06:02:16.252428+0000","last_unstale":"2024-04-16T06:02:17.719717+0000","last_undegraded":"2024-04-16T06:02:17.719717+0000","last_fullsized":"2024-04-16T06:02:17.719717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:15.201766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:15.201766+0000","last_clean_scrub_stamp":"2024-04-16T06:02:15.201766+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27044,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830556,"statfs":{"total":107374182400,"available":107346489344,"internally_reserved":0,"allocated":94208,"data_stored":21538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":611906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":611906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T06:02:19.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T06:02:19.387 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.386+0000 7f9cab62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.391 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.393+0000 7f9cab62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.574 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:19.575 INFO:teuthology.orchestra.run.smithi169.stderr:dumped all 2024-04-16T06:02:19.587 INFO:teuthology.orchestra.run.smithi169.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-04-16T06:02:19.140028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":314572800,"kb_used":82668,"kb_used_data":1436,"kb_used_omap":0,"kb_used_meta":80832,"kb_avail":314490132,"statfs":{"total":322122547200,"available":322037895168,"internally_reserved":0,"allocated":1470464,"data_stored":1245350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82771968},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.652488"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.355005+0000","last_change":"2024-04-16T06:02:17.356153+0000","last_active":"2024-04-16T06:02:17.355005+0000","last_peered":"2024-04-16T06:02:17.355005+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.349456+0000","last_became_peered":"2024-04-16T06:02:17.349456+0000","last_unstale":"2024-04-16T06:02:17.355005+0000","last_undegraded":"2024-04-16T06:02:17.355005+0000","last_fullsized":"2024-04-16T06:02:17.355005+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.336857+0000","last_change":"2024-04-16T06:02:17.338547+0000","last_active":"2024-04-16T06:02:17.336857+0000","last_peered":"2024-04-16T06:02:17.336857+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.333212+0000","last_became_peered":"2024-04-16T06:02:17.333212+0000","last_unstale":"2024-04-16T06:02:17.336857+0000","last_undegraded":"2024-04-16T06:02:17.336857+0000","last_fullsized":"2024-04-16T06:02:17.336857+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.331530+0000","last_change":"2024-04-16T06:02:17.332764+0000","last_active":"2024-04-16T06:02:17.331530+0000","last_peered":"2024-04-16T06:02:17.331530+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.325662+0000","last_became_peered":"2024-04-16T06:02:17.325662+0000","last_unstale":"2024-04-16T06:02:17.331530+0000","last_undegraded":"2024-04-16T06:02:17.331530+0000","last_fullsized":"2024-04-16T06:02:17.331530+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.316521+0000","last_change":"2024-04-16T06:02:17.317578+0000","last_active":"2024-04-16T06:02:17.316521+0000","last_peered":"2024-04-16T06:02:17.316521+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.308308+0000","last_became_peered":"2024-04-16T06:02:17.308308+0000","last_unstale":"2024-04-16T06:02:17.316521+0000","last_undegraded":"2024-04-16T06:02:17.316521+0000","last_fullsized":"2024-04-16T06:02:17.316521+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.332054+0000","last_change":"2024-04-16T06:02:17.333634+0000","last_active":"2024-04-16T06:02:17.332054+0000","last_peered":"2024-04-16T06:02:17.332054+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.327541+0000","last_became_peered":"2024-04-16T06:02:17.327541+0000","last_unstale":"2024-04-16T06:02:17.332054+0000","last_undegraded":"2024-04-16T06:02:17.332054+0000","last_fullsized":"2024-04-16T06:02:17.332054+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:18.256234+0000","last_change":"2024-04-16T06:02:17.344127+0000","last_active":"2024-04-16T06:02:18.256234+0000","last_peered":"2024-04-16T06:02:18.256234+0000","last_clean":"2024-04-16T06:02:18.256234+0000","last_became_active":"2024-04-16T06:02:17.339028+0000","last_became_peered":"2024-04-16T06:02:17.339028+0000","last_unstale":"2024-04-16T06:02:18.256234+0000","last_undegraded":"2024-04-16T06:02:18.256234+0000","last_fullsized":"2024-04-16T06:02:18.256234+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-16T06:02:18.253126+0000","last_change":"2024-04-16T06:02:17.325482+0000","last_active":"2024-04-16T06:02:18.253126+0000","last_peered":"2024-04-16T06:02:18.253126+0000","last_clean":"2024-04-16T06:02:18.253126+0000","last_became_active":"2024-04-16T06:02:17.319053+0000","last_became_peered":"2024-04-16T06:02:17.319053+0000","last_unstale":"2024-04-16T06:02:18.253126+0000","last_undegraded":"2024-04-16T06:02:18.253126+0000","last_fullsized":"2024-04-16T06:02:18.253126+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.320870+0000","last_change":"2024-04-16T06:02:17.322229+0000","last_active":"2024-04-16T06:02:17.320870+0000","last_peered":"2024-04-16T06:02:17.320870+0000","last_clean":"2024-04-16T06:02:16.205622+0000","last_became_active":"2024-04-16T06:02:17.317282+0000","last_became_peered":"2024-04-16T06:02:17.317282+0000","last_unstale":"2024-04-16T06:02:17.320870+0000","last_undegraded":"2024-04-16T06:02:17.320870+0000","last_fullsized":"2024-04-16T06:02:17.320870+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:16.205622+0000","last_clean_scrub_stamp":"2024-04-16T06:02:16.205622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'96","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-04-16T06:02:17.719717+0000","last_change":"2024-04-16T06:02:16.257389+0000","last_active":"2024-04-16T06:02:17.719717+0000","last_peered":"2024-04-16T06:02:17.719717+0000","last_clean":"2024-04-16T06:02:17.719717+0000","last_became_active":"2024-04-16T06:02:16.252428+0000","last_became_peered":"2024-04-16T06:02:16.252428+0000","last_unstale":"2024-04-16T06:02:17.719717+0000","last_undegraded":"2024-04-16T06:02:17.719717+0000","last_fullsized":"2024-04-16T06:02:17.719717+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-16T06:02:15.201766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-16T06:02:15.201766+0000","last_clean_scrub_stamp":"2024-04-16T06:02:15.201766+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27044,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104830556,"statfs":{"total":107374182400,"available":107346489344,"internally_reserved":0,"allocated":94208,"data_stored":21538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":611906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":104857600,"kb_used":27812,"kb_used_data":672,"kb_used_omap":0,"kb_used_meta":26944,"kb_avail":104829788,"statfs":{"total":107374182400,"available":107345702912,"internally_reserved":0,"allocated":688128,"data_stored":611906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27590656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-16T06:02:19.587 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-16T06:02:19.587 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-16T06:02:19.587 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-16T06:02:19.588 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-16T06:02:19.699 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.698+0000 7f5a4e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.702 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T06:02:19.705+0000 7f5a4e02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T06:02:19.929 INFO:teuthology.orchestra.run.smithi169.stdout: 2024-04-16T06:02:19.929 INFO:teuthology.orchestra.run.smithi169.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-16T06:02:19.942 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-16T06:02:19.943 INFO:teuthology.run_tasks:Running task workunit... 2024-04-16T06:02:19.955 INFO:tasks.workunit:Pulling workunits from ref c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:02:19.955 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-16T06:02:19.955 DEBUG:teuthology.orchestra.run.smithi169:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T06:02:19.971 INFO:teuthology.orchestra.run.smithi169.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-16T06:02:19.971 DEBUG:teuthology.orchestra.run.smithi169:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:20.026 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-16T06:02:20.026 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-16T06:02:20.081 INFO:tasks.workunit:timeout=3h 2024-04-16T06:02:20.081 INFO:tasks.workunit:cleanup=True 2024-04-16T06:02:20.082 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c0a287816afffa8ea7ac24bcf65ff9c48123b4cc 2024-04-16T06:02:20.138 INFO:tasks.workunit.client.0.smithi169.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr:Note: switching to 'c0a287816afffa8ea7ac24bcf65ff9c48123b4cc'. 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr:state without impacting any branches by switching back to a branch. 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.716 INFO:tasks.workunit.client.0.smithi169.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: git switch -c 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr:Or undo this operation with: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: git switch - 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr: 2024-04-16T06:03:40.717 INFO:tasks.workunit.client.0.smithi169.stderr:HEAD is now at c0a287816af seastar: submodule update 2024-04-16T06:03:40.722 DEBUG:teuthology.orchestra.run.smithi169:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-16T06:03:40.784 INFO:tasks.workunit.client.0.smithi169.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-16T06:03:40.786 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T06:03:40.786 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-16T06:03:41.475 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-16T06:03:41.547 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-16T06:03:41.589 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-16T06:03:41.591 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T06:03:41.591 INFO:tasks.workunit.client.0.smithi169.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-16T06:03:41.630 INFO:tasks.workunit.client.0.smithi169.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-16T06:03:41.634 DEBUG:teuthology.orchestra.run.smithi169:> set -ex 2024-04-16T06:03:41.634 DEBUG:teuthology.orchestra.run.smithi169:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-16T06:03:41.690 INFO:tasks.workunit:Running workunits matching rbd/crimson/test_crimson_librbd.sh on client.0... 2024-04-16T06:03:41.691 INFO:tasks.workunit:Running workunit rbd/crimson/test_crimson_librbd.sh... 2024-04-16T06:03:41.691 DEBUG:teuthology.orchestra.run.smithi169:workunit test rbd/crimson/test_crimson_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2024-04-16T06:03:41.822 INFO:tasks.workunit.client.0.smithi169.stdout:seed 34580 2024-04-16T06:03:41.823 INFO:tasks.workunit.client.0.smithi169.stdout:Note: Google Test filter = TestLibRBD.* 2024-04-16T06:03:41.823 INFO:tasks.workunit.client.0.smithi169.stdout:[==========] Running 168 tests from 1 test suite. 2024-04-16T06:03:41.823 INFO:tasks.workunit.client.0.smithi169.stdout:[----------] Global test environment set-up. 2024-04-16T06:03:41.823 INFO:tasks.workunit.client.0.smithi169.stdout:[----------] 168 tests from TestLibRBD 2024-04-16T06:03:41.868 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-04-16T06:03:43.312 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.498 INFO:tasks.workunit.client.0.smithi169.stdout:image has size 2097152 and order 22 2024-04-16T06:03:45.498 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.CreateAndStat (3630 ms) 2024-04-16T06:03:45.498 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-04-16T06:03:45.499 INFO:tasks.workunit.client.0.smithi169.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3085-gc0a28781/rpm/el9/BUILD/ceph-19.0.0-3085-gc0a28781/src/test/librbd/test_librbd.cc:780: Skipped 2024-04-16T06:03:45.499 INFO:tasks.workunit.client.0.smithi169.stdout:Skipping due to unmet REQUIRE 2024-04-16T06:03:45.499 INFO:tasks.workunit.client.0.smithi169.stdout:[ SKIPPED ] TestLibRBD.CreateWithSameDataPool (0 ms) 2024-04-16T06:03:45.499 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-04-16T06:03:45.499 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.582 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.CreateAndStatPP (84 ms) 2024-04-16T06:03:45.582 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.GetId 2024-04-16T06:03:45.582 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.660 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.GetId (78 ms) 2024-04-16T06:03:45.660 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-04-16T06:03:45.660 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.746 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.GetIdPP (86 ms) 2024-04-16T06:03:45.746 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-04-16T06:03:45.746 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.823 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (76 ms) 2024-04-16T06:03:45.823 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-04-16T06:03:45.823 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (80 ms) 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3085-gc0a28781/rpm/el9/BUILD/ceph-19.0.0-3085-gc0a28781/src/test/librbd/test_librbd.cc:939: Skipped 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:Skipping due to unmet REQUIRE 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:[ SKIPPED ] TestLibRBD.TestGetCreateTimestamp (0 ms) 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-04-16T06:03:45.902 INFO:tasks.workunit.client.0.smithi169.stdout:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-3085-gc0a28781/rpm/el9/BUILD/ceph-19.0.0-3085-gc0a28781/src/test/librbd/test_librbd.cc:962: Skipped 2024-04-16T06:03:45.903 INFO:tasks.workunit.client.0.smithi169.stdout:Skipping due to unmet REQUIRE 2024-04-16T06:03:45.903 INFO:tasks.workunit.client.0.smithi169.stdout:[ SKIPPED ] TestLibRBD.GetCreateTimestampPP (0 ms) 2024-04-16T06:03:45.903 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.OpenAio 2024-04-16T06:03:45.903 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:45.978 INFO:tasks.workunit.client.0.smithi169.stdout:image has size 2097152 and order 22 2024-04-16T06:03:45.978 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.OpenAio (76 ms) 2024-04-16T06:03:45.978 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-04-16T06:03:45.989 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.OpenAioFail (11 ms) 2024-04-16T06:03:45.990 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-04-16T06:03:45.990 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:46.112 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.OpenAioPP (123 ms) 2024-04-16T06:03:46.112 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-04-16T06:03:46.123 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.OpenAioFailPP (11 ms) 2024-04-16T06:03:46.123 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-04-16T06:03:46.124 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:46.317 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.ResizeAndStat (194 ms) 2024-04-16T06:03:46.317 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-04-16T06:03:46.317 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:46.471 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (154 ms) 2024-04-16T06:03:46.472 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-04-16T06:03:46.472 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:46.662 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (190 ms) 2024-04-16T06:03:46.662 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-04-16T06:03:46.662 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:03:46.850 INFO:tasks.workunit.client.0.smithi169.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (189 ms) 2024-04-16T06:03:46.850 INFO:tasks.workunit.client.0.smithi169.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-04-16T06:03:51.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-16T06:03:56.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-04-16T06:04:02.015 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-04-16T06:04:07.418 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-04-16T06:04:12.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-04-16T06:04:18.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-04-16T06:04:19.636 INFO:tasks.workunit.client.0.smithi169.stdout:using old format 2024-04-16T06:04:23.627 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-04-16T06:04:29.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2024-04-16T06:04:34.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2024-04-16T06:04:39.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-04-16T06:04:45.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2024-04-16T06:04:50.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2024-04-16T06:04:56.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-04-16T06:05:01.448 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-04-16T06:05:06.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2024-04-16T06:05:12.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2024-04-16T06:05:17.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2024-04-16T06:05:21.160 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:21.161+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:23.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-04-16T06:05:26.160 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:26.162+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:28.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2024-04-16T06:05:31.161 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:31.163+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:33.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~103s 2024-04-16T06:05:36.161 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:36.163+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:39.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~108s 2024-04-16T06:05:41.162 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:41.164+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:44.671 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~113s 2024-04-16T06:05:46.162 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:46.164+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:50.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~119s 2024-04-16T06:05:51.163 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:51.165+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:05:55.478 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~124s 2024-04-16T06:05:56.164 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:05:56.165+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:00.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-04-16T06:06:01.164 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:01.166+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:06.165 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:06.167+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:06.285 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-04-16T06:06:11.166 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:11.168+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:11.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-04-16T06:06:16.166 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:16.167+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:17.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-04-16T06:06:21.167 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:21.169+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:22.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-04-16T06:06:26.168 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:26.170+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:27.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~157s 2024-04-16T06:06:31.168 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:31.170+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:33.298 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~162s 2024-04-16T06:06:36.169 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:36.171+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:38.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~167s 2024-04-16T06:06:41.169 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:41.171+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:44.104 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~173s 2024-04-16T06:06:46.170 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:46.172+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:49.508 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~178s 2024-04-16T06:06:51.171 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:51.173+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:06:54.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~184s 2024-04-16T06:06:56.171 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:06:56.173+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:00.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~189s 2024-04-16T06:07:01.172 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:01.174+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:05.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-04-16T06:07:06.173 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:06.173+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:11.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-04-16T06:07:11.173 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:11.175+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:16.174 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:16.176+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:16.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-04-16T06:07:21.174 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:21.176+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:21.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~211s 2024-04-16T06:07:26.175 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:26.177+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:27.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~216s 2024-04-16T06:07:31.176 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:31.178+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:32.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~222s 2024-04-16T06:07:36.176 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:36.178+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:38.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~227s 2024-04-16T06:07:41.177 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:41.179+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:43.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~232s 2024-04-16T06:07:46.178 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:46.179+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:48.941 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~238s 2024-04-16T06:07:51.178 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:51.180+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:54.343 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~243s 2024-04-16T06:07:56.179 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:07:56.181+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:07:59.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2024-04-16T06:08:01.179 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:01.181+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:05.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~254s 2024-04-16T06:08:06.180 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:06.182+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:10.552 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2024-04-16T06:08:11.180 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:11.182+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:15.954 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-04-16T06:08:16.181 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:16.183+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:21.181 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:21.183+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:21.357 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-04-16T06:08:26.182 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:26.184+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:26.760 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-04-16T06:08:31.183 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:31.185+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:32.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-04-16T06:08:36.184 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:36.186+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:37.565 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-04-16T06:08:41.184 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:41.186+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:42.968 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2024-04-16T06:08:46.185 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:46.187+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:48.372 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2024-04-16T06:08:51.186 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:51.188+0000 7fdd8d5f9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.169:6805/1397559365 for 29sec e22 v22) 2024-04-16T06:08:53.774 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2024-04-16T06:08:53.774 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-04-16T06:08:54.177 INFO:tasks.ceph.osd.0:Sent signal 15 2024-04-16T06:08:54.178 INFO:tasks.ceph.osd.2:Sent signal 15 2024-04-16T06:08:54.178 INFO:tasks.ceph.mon.a:Sent signal 15 2024-04-16T06:08:54.178 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-04-16T06:08:54.179 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:54.181+0000 7fdd905ff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 31972) UID: 0 2024-04-16T06:08:54.179 INFO:tasks.ceph.mon.a.smithi169.stderr:2024-04-16T06:08:54.181+0000 7fdd905ff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-16T06:08:54.179 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:08:54.181+0000 7f6e741f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 31973) UID: 0 2024-04-16T06:08:54.179 INFO:tasks.ceph.mgr.x.smithi169.stderr:2024-04-16T06:08:54.181+0000 7f6e741f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-16T09:03:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T09:03:41.776 INFO:tasks.workunit:Stopping ['rbd/crimson/test_crimson_librbd.sh'] on client.0... 2024-04-16T09:03:41.777 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-16T09:03:42.052 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-04-16T09:03:42.241 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e3305dbfbea14ca0afd5a29fac152189 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-04-16T09:03:42.243 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-16T09:03:42.252 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-16T09:03:42.253 DEBUG:teuthology.orchestra.run.smithi169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-16T09:03:42.368 INFO:teuthology.orchestra.run.smithi169.stderr:2024-04-16T09:03:42.366+0000 7efff0b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-16T09:05:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-16T09:05:42.291 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:05:42.302 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-16T09:05:42.303 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-16T09:05:42.303 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-16T09:05:42.303 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:05:42.359 INFO:tasks.ceph.osd.0:Stopped 2024-04-16T09:05:42.359 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-16T09:05:42.359 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:05:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:05:42.359 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log' 2024-04-16T09:05:42.368 INFO:tasks.ceph.osd.1:Stopped 2024-04-16T09:05:42.368 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-16T09:05:42.369 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:05:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:05:42.370 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-04-16T09:05:42.379 INFO:tasks.ceph.osd.2:Stopped 2024-04-16T09:05:42.379 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-16T09:05:42.379 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-16T09:05:42.379 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:05:42.379 INFO:tasks.ceph.mgr.x:Stopped 2024-04-16T09:05:42.380 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-16T09:05:42.380 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-16T09:05:42.380 INFO:teuthology.orchestra.run:waiting for 300 2024-04-16T09:05:42.380 INFO:tasks.ceph.mon.a:Stopped 2024-04-16T09:05:42.380 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-16T09:05:42.381 DEBUG:teuthology.orchestra.run.smithi169:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'is full \(reached quota' | egrep -v '\(POOL_FULL\)' | head -n 1 2024-04-16T09:05:42.412 INFO:teuthology.orchestra.run.smithi169.stdout:2024-04-16T06:04:18.407509+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-04-16T09:05:42.413 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-04-16T09:05:42.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T09:05:42.413 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-16T09:05:42.511 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T09:05:42.512 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-16T09:05:42.584 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T09:05:42.584 DEBUG:teuthology.orchestra.run.smithi169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-16T09:05:42.645 INFO:tasks.ceph:Archiving mon data... 2024-04-16T09:05:42.646 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937/data/mon.a.tgz 2024-04-16T09:05:42.646 DEBUG:teuthology.orchestra.run.smithi169:> mktemp 2024-04-16T09:05:42.661 INFO:teuthology.orchestra.run.smithi169.stdout:/tmp/tmp.yjhhQ5LLl4 2024-04-16T09:05:42.662 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.yjhhQ5LLl4 2024-04-16T09:05:42.784 DEBUG:teuthology.orchestra.run.smithi169:> sudo chmod 0666 /tmp/tmp.yjhhQ5LLl4 2024-04-16T09:05:42.871 DEBUG:teuthology.orchestra.remote:smithi169:/tmp/tmp.yjhhQ5LLl4 is 227KB 2024-04-16T09:05:42.968 DEBUG:teuthology.orchestra.run.smithi169:> rm -fr /tmp/tmp.yjhhQ5LLl4 2024-04-16T09:05:42.983 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-16T09:05:42.984 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-16T09:05:43.108 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-16T09:05:43.108 INFO:tasks.ceph:Archiving crash dumps... 2024-04-16T09:05:43.110 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/lib/ceph/crash to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937/remote/smithi169/crash 2024-04-16T09:05:43.111 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-16T09:05:43.176 INFO:tasks.ceph:Compressing logs... 2024-04-16T09:05:43.176 DEBUG:teuthology.orchestra.run.smithi169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-16T09:05:43.261 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34277.log 2024-04-16T09:05:43.261 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32064.log 2024-04-16T09:05:43.261 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34277.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-16T09:05:43.261 INFO:teuthology.orchestra.run.smithi169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34277.log.gz 2024-04-16T09:05:43.261 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34065.log 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32064.log.gz 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34319.log 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34190.log 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34065.log.gz 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34319.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34319.log.gz 2024-04-16T09:05:43.262 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33983.log 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34190.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28732.log 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34190.log.gz 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.33983.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33983.log.gz 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.tmp-client.admin.28732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32218.log 2024-04-16T09:05:43.263 INFO:teuthology.orchestra.run.smithi169.stderr: 67.5% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28732.log.gz 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34151.log 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr: 85.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32454.log 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.32218.log.gz 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34870.log 2024-04-16T09:05:43.264 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34151.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34151.log.gz 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32454.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32454.log.gz 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33838.log 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.34870.log.gz 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33740.log 2024-04-16T09:05:43.265 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.33838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31976.log 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33838.log.gz 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.33740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33941.log 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33740.log.gz 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.31976.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31976.log.gz 2024-04-16T09:05:43.266 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.33941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32175.log 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33941.log.gz 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32263.log 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32175.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32175.log.gz 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34064.log 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32263.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32263.log.gz 2024-04-16T09:05:43.267 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34064.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34064.log.gz 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.33899.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr: /var/log/ceph/ceph-client.admin.33796.log 2024-04-16T09:05:43.268 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32422.log 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.33796.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33796.log.gz 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34063.log 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.32422.log: gzip -5 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32422.log.gz 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34580.log 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34063.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34063.log.gz 2024-04-16T09:05:43.269 INFO:teuthology.orchestra.run.smithi169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34191.log 2024-04-16T09:05:43.270 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34361.log 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34580.log.gz 2024-04-16T09:05:43.270 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T09:05:43.270 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34191.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34191.log.gz 2024-04-16T09:05:43.270 INFO:teuthology.orchestra.run.smithi169.stderr:/var/log/ceph/ceph-client.admin.34361.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34361.log.gz 2024-04-16T09:05:43.289 INFO:teuthology.orchestra.run.smithi169.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-16T09:05:43.325 INFO:teuthology.orchestra.run.smithi169.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-16T09:05:43.348 INFO:teuthology.orchestra.run.smithi169.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-16T09:05:43.350 INFO:teuthology.orchestra.run.smithi169.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-16T09:05:43.355 INFO:teuthology.orchestra.run.smithi169.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-16T09:05:43.358 INFO:teuthology.orchestra.run.smithi169.stderr: 2024-04-16T09:05:43.358 INFO:teuthology.orchestra.run.smithi169.stderr:real 0m0.126s 2024-04-16T09:05:43.358 INFO:teuthology.orchestra.run.smithi169.stderr:user 0m0.420s 2024-04-16T09:05:43.358 INFO:teuthology.orchestra.run.smithi169.stderr:sys 0m0.050s 2024-04-16T09:05:43.359 INFO:tasks.ceph:Archiving logs... 2024-04-16T09:05:43.359 DEBUG:teuthology.misc:Transferring archived files from smithi169:/var/log/ceph to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937/remote/smithi169/log 2024-04-16T09:05:43.360 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-16T09:05:43.565 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:05:43.577 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-16T09:05:43.591 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:05:43.595 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-16T09:05:43.595 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-16T09:05:43.653 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-16T09:05:43.661 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-16T09:05:43.661 DEBUG:teuthology.orchestra.run.smithi169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-16T09:05:43.680 INFO:teuthology.orchestra.run.smithi169.stderr:bash: line 1: ntpq: command not found 2024-04-16T09:05:43.685 INFO:teuthology.orchestra.run.smithi169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-16T09:05:43.685 INFO:teuthology.orchestra.run.smithi169.stdout:=============================================================================== 2024-04-16T09:05:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:^* hv01.front.sepia.ceph.com 3 7 377 28 +1187us[+1197us] +/- 48ms 2024-04-16T09:05:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 38 -1527us[-1517us] +/- 73ms 2024-04-16T09:05:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -246us[ -236us] +/- 68ms 2024-04-16T09:05:43.686 INFO:teuthology.orchestra.run.smithi169.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-04-16T09:05:43.686 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-16T09:05:43.695 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-16T09:05:43.696 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-16T09:05:43.704 DEBUG:teuthology.orchestra.run.smithi169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-16T09:05:43.793 DEBUG:teuthology.orchestra.run.smithi169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-16T09:05:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:05:43.824 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-16T09:05:43.832 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-16T09:05:43.840 INFO:teuthology.task.internal:Duration was 11495.615570 seconds 2024-04-16T09:05:43.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-16T09:05:43.847 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-16T09:05:43.848 DEBUG:teuthology.orchestra.run.smithi169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-16T09:05:43.909 INFO:teuthology.orchestra.run.smithi169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-16T09:05:44.185 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-16T09:05:44.185 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi169.front.sepia.ceph.com 2024-04-16T09:05:44.185 DEBUG:teuthology.orchestra.run.smithi169:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-16T09:05:44.205 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-16T09:05:44.206 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-16T09:05:44.276 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-16T09:05:44.276 DEBUG:teuthology.orchestra.run.smithi169:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-16T09:05:44.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-16T09:05:44.435 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-16T09:05:44.435 DEBUG:teuthology.orchestra.run.smithi169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-16T09:05:44.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-16T09:05:44.474 DEBUG:teuthology.orchestra.run.smithi169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-16T09:05:44.531 INFO:teuthology.orchestra.run.smithi169.stdout:kernel.core_pattern = core 2024-04-16T09:05:44.550 DEBUG:teuthology.orchestra.run.smithi169:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-16T09:05:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:05:44.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-16T09:05:44.615 INFO:teuthology.task.internal:Transferring archived files... 2024-04-16T09:05:44.615 DEBUG:teuthology.misc:Transferring archived files from smithi169:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937/remote/smithi169 2024-04-16T09:05:44.616 DEBUG:teuthology.orchestra.run.smithi169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-16T09:05:44.682 INFO:teuthology.task.internal:Removing archive directory... 2024-04-16T09:05:44.683 DEBUG:teuthology.orchestra.run.smithi169:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-16T09:05:44.732 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-16T09:05:44.743 INFO:teuthology.task.internal:Not uploading archives. 2024-04-16T09:05:44.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-16T09:05:44.752 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-16T09:05:44.752 DEBUG:teuthology.orchestra.run.smithi169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-16T09:05:44.787 INFO:teuthology.orchestra.run.smithi169.stdout: 265317 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 09:05 /home/ubuntu/cephtest 2024-04-16T09:05:44.788 INFO:teuthology.orchestra.run.smithi169.stdout: 403682 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 06:02 /home/ubuntu/cephtest/mnt.0 2024-04-16T09:05:44.788 INFO:teuthology.orchestra.run.smithi169.stdout: 403683 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Apr 16 06:03 /home/ubuntu/cephtest/mnt.0/client.0 2024-04-16T09:05:44.788 INFO:teuthology.orchestra.run.smithi169.stdout: 418052 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 16 06:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-16T09:05:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-16T09:05:44.789 INFO:teuthology.orchestra.run.smithi169.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-04-16T09:05:44.789 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2822, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2654, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 2420, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1656, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_c0a287816afffa8ea7ac24bcf65ff9c48123b4cc/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi169 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-04-16T09:05:44.801 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-16T09:05:44.811 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-16T09:05:44.820 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-04-16T09:05:44.820 INFO:teuthology.nuke:Checking targets against current locks 2024-04-16T09:05:44.845 DEBUG:teuthology.nuke:shortname: smithi169 2024-04-16T09:05:44.846 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-16T09:05:44.867 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi169.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-16_05:45:04-crimson-rados:rbd-wip-nitzan-seastore-re-enable-gate-debug-flag-distro-default-smithi/7657937', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-16 05:46:05.681535', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKaUGh9tzksP6/9SThjKLXdcAk18nfXCKPCFuUruX739LRCFwz928P9bstiBxnORvfJXtxjYL/l+Hyb1IrYXIVs='} 2024-04-16T09:05:44.913 INFO:teuthology.orchestra.console.smithi169:Power off 2024-04-16T09:05:44.913 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-16T09:05:44.938 DEBUG:teuthology.orchestra.console.smithi169:power off output: Chassis Power Control: Down/Off 2024-04-16T09:05:44.939 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:05:44.961 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-16T09:05:48.962 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:05:49.071 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-16T09:05:53.076 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:05:53.188 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is on 2024-04-16T09:05:57.193 DEBUG:teuthology.orchestra.console.smithi169:pexpect command: ipmitool -H smithi169.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-16T09:05:57.304 DEBUG:teuthology.orchestra.console.smithi169:check power output: Chassis Power is off 2024-04-16T09:05:57.405 INFO:teuthology.orchestra.console.smithi169:Power off completed 2024-04-16T09:05:57.512 INFO:teuthology.run:Summary data: description: crimson-rados:rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} duration: 11495.61557006836 failure_reason: 'Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi169 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0a287816afffa8ea7ac24bcf65ff9c48123b4cc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh''' flavor: crimson owner: scheduled_nmordech@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e3305dbfbea14ca0afd5a29fac152189 status: fail success: false 2024-04-16T09:05:57.512 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-16T09:05:57.608 INFO:teuthology.run:FAIL