2024-03-27T01:56:30.908 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-27T01:56:30.908 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} --name yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi --archive /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043 --owner scheduled_yingxin@teuthology --verbose -- /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/orig.config.yaml 2024-03-27T01:56:30.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T01:56:31.036 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043 branch: ci-yingxin-crimson-load-obc-2 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7625043' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-2 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 4254c92712efa5f9025679283afe440af32d0589 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 4254c92712efa5f9025679283afe440af32d0589 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-2 env: CRIMSON_COMPAT: '1' sha1: 4254c92712efa5f9025679283afe440af32d0589 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4254c92712efa5f9025679283afe440af32d0589 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_4254c92712efa5f9025679283afe440af32d0589/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 4254c92712efa5f9025679283afe440af32d0589 targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGOCRXDIc5f5ZrdgQ43O5wUWTbQPI4Eq8i7O2NCIzQJPCON69xoODrNGjrxYrIAZcxP7Wj9o9smcQmSDZoLT7Y= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_01:32:53 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-27T01:56:31.036 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_4254c92712efa5f9025679283afe440af32d0589/qa; will attempt to use it 2024-03-27T01:56:31.037 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_4254c92712efa5f9025679283afe440af32d0589/qa/tasks 2024-03-27T01:56:31.037 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-27T01:56:31.039 INFO:teuthology.task.internal:Checking packages... 2024-03-27T01:56:31.063 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4254c92712efa5f9025679283afe440af32d0589' 2024-03-27T01:56:31.063 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-27T01:56:31.063 INFO:teuthology.packaging:ref: None 2024-03-27T01:56:31.063 INFO:teuthology.packaging:tag: None 2024-03-27T01:56:31.063 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-2 2024-03-27T01:56:31.063 INFO:teuthology.packaging:sha1: 4254c92712efa5f9025679283afe440af32d0589 2024-03-27T01:56:31.063 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-2 2024-03-27T01:56:31.284 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2463.g4254c927 2024-03-27T01:56:31.285 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-27T01:56:31.291 INFO:teuthology.task.internal:no buildpackages task found 2024-03-27T01:56:31.292 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-27T01:56:31.335 INFO:teuthology.task.internal:Saving configuration 2024-03-27T01:56:31.352 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-27T01:56:31.360 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-27T01:56:31.382 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-27 01:52:04.453956', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGOCRXDIc5f5ZrdgQ43O5wUWTbQPI4Eq8i7O2NCIzQJPCON69xoODrNGjrxYrIAZcxP7Wj9o9smcQmSDZoLT7Y='} 2024-03-27T01:56:31.382 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-27T01:56:31.388 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-27T01:56:31.388 INFO:teuthology.run_tasks:Running task console_log... 2024-03-27T01:56:31.443 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fea7ed26f70>, signals=[15]) 2024-03-27T01:56:31.444 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-27T01:56:31.450 INFO:teuthology.task.internal:Opening connections... 2024-03-27T01:56:31.450 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T01:56:31.452 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T01:56:31.555 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-27T01:56:31.561 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-27T01:56:31.628 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-27T01:56:31.628 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-27T01:56:31.686 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-27T01:56:31.687 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-27T01:56:31.687 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-27T01:56:31.687 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-27T01:56:31.687 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-27T01:56:31.687 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-27T01:56:31.719 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-27T01:56:31.729 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-27T01:56:31.772 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-27T01:56:31.773 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-27T01:56:31.790 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-27T01:56:31.801 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-27T01:56:31.801 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-27T01:56:31.854 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-27T01:56:31.885 INFO:teuthology.run_tasks:Running task kernel... 2024-03-27T01:56:31.898 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-27T01:56:31.898 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-27T01:56:31.899 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-27T01:56:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T01:56:31.916 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-27T01:56:31.973 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-03-27T01:56:31.973 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-03-27T01:56:31.973 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-03-27T01:56:34.155 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-27T01:56:35.696 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-27T01:56:39.040 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-27T01:56:42.587 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 7.7 MB/s | 19 MB 00:02 2024-03-27T01:56:49.154 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 2.1 MB/s | 6.1 MB 00:02 2024-03-27T01:56:50.780 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-27T01:56:52.850 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-27T01:56:58.048 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-27T01:57:00.313 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-27T01:57:00.313 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-27T01:57:00.405 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T01:57:00.411 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-03-27T01:57:00.412 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:00.412 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 76 M 2024-03-27T01:57:00.412 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-03-27T01:57:00.412 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T01:57:00.932 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 177 kB/s | 66 kB 00:00 2024-03-27T01:57:04.941 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 7.6 MB/s | 33 MB 00:04 2024-03-27T01:57:06.148 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 2.7 MB/s | 15 MB 00:05 2024-03-27T01:57:07.957 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 3.9 MB/s | 27 MB 00:07 2024-03-27T01:57:07.957 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T01:57:07.957 INFO:teuthology.orchestra.run.smithi033.stdout:Total 10 MB/s | 76 MB 00:07 2024-03-27T01:57:08.652 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T01:57:09.075 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T01:57:09.075 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T01:57:10.125 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T01:57:10.126 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T01:57:12.705 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T01:57:13.357 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-27T01:57:13.507 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T01:57:16.394 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T01:57:16.526 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T01:57:21.633 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T01:57:21.676 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:26.801 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:57.593 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:57.593 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-27T01:57:57.593 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:57.638 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:58.094 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:58.094 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-27T01:57:58.095 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-27T01:57:58.095 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-27T01:57:58.529 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:58.530 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T01:57:58.873 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T01:57:58.977 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T01:57:58.978 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T01:57:58.978 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-27T01:57:58.978 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-27T01:57:58.978 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-27T01:57:58.978 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-27T01:57:58.979 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-03-27T01:57:58.979 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-27T01:57:59.061 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-27T01:57:59.061 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-27T01:57:59.062 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-03-27T01:57:59.114 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-27T01:57:59.115 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T01:57:59.116 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-03-27T01:57:59.117 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-27T01:57:59.117 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-432.el9.x86_64 2024-03-27T01:57:59.117 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-27T01:57:59.656 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-03-27T01:58:01.079 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-27T01:58:01.116 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-03-27T01:58:01.118 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-27T01:58:01.134 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.ZXEYq1tCIe 2024-03-27T01:58:01.134 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZXEYq1tCIe 2024-03-27T01:58:01.206 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.ZXEYq1tCIe 2024-03-27T01:58:01.342 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.ZXEYq1tCIe is 6KB 2024-03-27T01:58:01.363 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.ZXEYq1tCIe 2024-03-27T01:58:01.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-03-27T01:58:01.445 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-27T01:58:01.445 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-27T01:58:01.445 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-27T01:58:01.445 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-27T01:58:01.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-27T01:58:01.622 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-03-27T01:58:31.653 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-27T01:58:31.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T01:58:31.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T01:58:50.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-27T01:58:59.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T01:58:59.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T01:59:17.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-27T01:59:29.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T01:59:29.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T01:59:30.624 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-27T01:59:31.108 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T01:59:31.109 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-27T01:59:31.109 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-27T01:59:31.124 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-432.el9.x86_64 2024-03-27T01:59:31.124 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-27T01:59:31.124 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-27T01:59:31.124 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-27T01:59:32.125 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-27T01:59:32.125 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-27T01:59:32.185 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-03-27T01:59:32.233 DEBUG:teuthology.parallel:result is None 2024-03-27T01:59:32.234 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-27T01:59:32.241 INFO:teuthology.task.internal:Creating test directory... 2024-03-27T01:59:32.241 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-27T01:59:32.255 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-27T01:59:32.290 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-27T01:59:32.338 INFO:teuthology.task.internal:Creating archive directory... 2024-03-27T01:59:32.339 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-27T01:59:32.358 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-27T01:59:32.388 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-27T01:59:32.388 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-27T01:59:32.426 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T01:59:32.439 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-27T01:59:32.441 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-27T01:59:32.448 INFO:teuthology.task.internal:Configuring sudo... 2024-03-27T01:59:32.448 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-27T01:59:32.508 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-27T01:59:32.517 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-27T01:59:32.517 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-27T01:59:32.564 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T01:59:32.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-27T01:59:32.687 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T01:59:32.768 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-27T01:59:32.810 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T01:59:32.810 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-27T01:59:32.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-27T01:59:32.976 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T01:59:33.082 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-27T01:59:33.099 INFO:teuthology.task.internal:Starting timer... 2024-03-27T01:59:33.099 INFO:teuthology.run_tasks:Running task pcp... 2024-03-27T01:59:33.142 INFO:teuthology.run_tasks:Running task selinux... 2024-03-27T01:59:33.191 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-27T01:59:33.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-27T01:59:33.250 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-27T01:59:33.252 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-27T01:59:33.252 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-27T01:59:33.270 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-27T01:59:33.270 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive'} 2024-03-27T01:59:33.270 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T01:59:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T01:59:33.339 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-27T01:59:33.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-27T01:59:33.414 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-27T01:59:33.423 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-27T01:59:33.440 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-27T01:59:33.442 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-27T02:03:50.901 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2024-03-27T02:03:50.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T02:03:50.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-27T02:03:50.998 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-27T02:03:51.077 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-27T02:03:51.077 INFO:teuthology.run_tasks:Running task clock... 2024-03-27T02:03:51.089 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-27T02:03:51.089 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-27T02:03:51.089 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T02:03:51.147 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-27T02:03:51.164 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-27T02:03:51.193 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-27T02:03:51.207 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-27T02:03:51.223 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-27T02:03:51.240 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-27T02:03:51.300 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T02:03:51.304 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T02:03:51.305 INFO:teuthology.run_tasks:Running task install... 2024-03-27T02:03:51.313 DEBUG:teuthology.task.install:project ceph 2024-03-27T02:03:51.314 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4254c92712efa5f9025679283afe440af32d0589'}} 2024-03-27T02:03:51.314 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4254c92712efa5f9025679283afe440af32d0589'} 2024-03-27T02:03:51.314 INFO:teuthology.task.install:Using flavor: crimson 2024-03-27T02:03:51.320 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-27T02:03:51.320 INFO:teuthology.task.install:extra packages: [] 2024-03-27T02:03:51.320 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4254c92712efa5f9025679283afe440af32d0589', 'tag': None, 'wait_for_package': False} 2024-03-27T02:03:51.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4254c92712efa5f9025679283afe440af32d0589 2024-03-27T02:03:51.473 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-2/4254c92712efa5f9025679283afe440af32d0589/centos/9/flavors/crimson/ 2024-03-27T02:03:51.473 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2463.g4254c927 2024-03-27T02:03:51.645 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-2/4254c92712efa5f9025679283afe440af32d0589/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-2/4254c92712efa5f9025679283afe440af32d0589/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-2/4254c92712efa5f9025679283afe440af32d0589/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-27T02:03:51.645 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:03:51.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-27T02:03:51.673 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-27T02:03:51.673 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4254c92712efa5f9025679283afe440af32d0589/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-27T02:03:51.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-27T02:03:51.821 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-27T02:03:51.888 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-03-27T02:03:51.889 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-03-27T02:03:52.146 INFO:teuthology.orchestra.run.smithi033.stdout:74 files removed 2024-03-27T02:03:52.172 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw 2024-03-27T02:03:52.855 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 230 kB/s | 87 kB 00:00 2024-03-27T02:03:53.289 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-03-27T02:03:53.639 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 6.6 kB/s | 2.0 kB 00:00 2024-03-27T02:03:54.126 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-27T02:03:55.634 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-27T02:03:57.053 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-03-27T02:03:59.501 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-27T02:04:12.368 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 681 kB/s | 6.1 MB 00:09 2024-03-27T02:04:13.997 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-27T02:04:15.528 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-27T02:04:20.569 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-27T02:04:23.143 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-2463.g4254c927.el9 ceph 9.6 M 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-2463.g4254c927.el9 ceph 3.4 M 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-2463.g4254c927.el9 ceph 3.7 M 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:04:23.154 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-2463.g4254c927.el9 ceph 5.9 M 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-2463.g4254c927.el9 ceph 19 M 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-2463.g4254c927.el9 ceph 25 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-2463.g4254c927.el9 ceph 796 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-27T02:04:23.155 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-2463.g4254c927.el9 ceph 548 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-2463.g4254c927.el9 ceph 5.0 M 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 2:19.0.0-2463.g4254c927.el9 ceph 45 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-2463.g4254c927.el9 ceph 134 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-2463.g4254c927.el9 ceph 159 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-2463.g4254c927.el9 ceph 315 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-2463.g4254c927.el9 ceph 292 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-2463.g4254c927.el9 ceph 99 k 2024-03-27T02:04:23.156 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:04:23.157 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:23.158 INFO:teuthology.orchestra.run.smithi033.stdout:Install 30 Packages 2024-03-27T02:04:23.158 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 2 Packages 2024-03-27T02:04:23.158 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:23.158 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 58 M 2024-03-27T02:04:23.158 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:04:25.070 INFO:teuthology.orchestra.run.smithi033.stdout:(1/32): ceph-radosgw-19.0.0-2463.g4254c927.el9. 5.8 MB/s | 9.6 MB 00:01 2024-03-27T02:04:25.137 INFO:teuthology.orchestra.run.smithi033.stdout:(2/32): ceph-selinux-19.0.0-2463.g4254c927.el9. 378 kB/s | 25 kB 00:00 2024-03-27T02:04:25.221 INFO:teuthology.orchestra.run.smithi033.stdout:(3/32): ceph-base-19.0.0-2463.g4254c927.el9.x86 3.3 MB/s | 5.9 MB 00:01 2024-03-27T02:04:25.296 INFO:teuthology.orchestra.run.smithi033.stdout:(4/32): libcephfs2-19.0.0-2463.g4254c927.el9.x8 4.9 MB/s | 796 kB 00:00 2024-03-27T02:04:25.321 INFO:teuthology.orchestra.run.smithi033.stdout:(5/32): libradosstriper1-19.0.0-2463.g4254c927. 5.3 MB/s | 548 kB 00:00 2024-03-27T02:04:25.380 INFO:teuthology.orchestra.run.smithi033.stdout:(6/32): python3-ceph-argparse-19.0.0-2463.g4254 772 kB/s | 45 kB 00:00 2024-03-27T02:04:25.464 INFO:teuthology.orchestra.run.smithi033.stdout:(7/32): python3-ceph-common-19.0.0-2463.g4254c9 1.6 MB/s | 134 kB 00:00 2024-03-27T02:04:25.547 INFO:teuthology.orchestra.run.smithi033.stdout:(8/32): python3-cephfs-19.0.0-2463.g4254c927.el 1.9 MB/s | 159 kB 00:00 2024-03-27T02:04:25.664 INFO:teuthology.orchestra.run.smithi033.stdout:(9/32): librgw2-19.0.0-2463.g4254c927.el9.x86_6 14 MB/s | 5.0 MB 00:00 2024-03-27T02:04:25.689 INFO:teuthology.orchestra.run.smithi033.stdout:(10/32): python3-rados-19.0.0-2463.g4254c927.el 2.2 MB/s | 315 kB 00:00 2024-03-27T02:04:25.806 INFO:teuthology.orchestra.run.smithi033.stdout:(11/32): python3-rbd-19.0.0-2463.g4254c927.el9. 2.0 MB/s | 292 kB 00:00 2024-03-27T02:04:25.832 INFO:teuthology.orchestra.run.smithi033.stdout:(12/32): python3-rgw-19.0.0-2463.g4254c927.el9. 699 kB/s | 99 kB 00:00 2024-03-27T02:04:25.965 INFO:teuthology.orchestra.run.smithi033.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-27T02:04:25.999 INFO:teuthology.orchestra.run.smithi033.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 954 kB/s | 159 kB 00:00 2024-03-27T02:04:26.016 INFO:teuthology.orchestra.run.smithi033.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-27T02:04:26.141 INFO:teuthology.orchestra.run.smithi033.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 661 kB 00:00 2024-03-27T02:04:26.167 INFO:teuthology.orchestra.run.smithi033.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-27T02:04:26.183 INFO:teuthology.orchestra.run.smithi033.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-27T02:04:26.276 INFO:teuthology.orchestra.run.smithi033.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 391 kB/s | 36 kB 00:00 2024-03-27T02:04:26.318 INFO:teuthology.orchestra.run.smithi033.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-03-27T02:04:26.343 INFO:teuthology.orchestra.run.smithi033.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-03-27T02:04:26.751 INFO:teuthology.orchestra.run.smithi033.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 712 kB/s | 309 kB 00:00 2024-03-27T02:04:26.827 INFO:teuthology.orchestra.run.smithi033.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 346 kB/s | 26 kB 00:00 2024-03-27T02:04:26.894 INFO:teuthology.orchestra.run.smithi033.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-03-27T02:04:26.977 INFO:teuthology.orchestra.run.smithi033.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 808 kB/s | 67 kB 00:00 2024-03-27T02:04:27.053 INFO:teuthology.orchestra.run.smithi033.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.2 MB/s | 4.4 MB 00:00 2024-03-27T02:04:27.128 INFO:teuthology.orchestra.run.smithi033.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-27T02:04:27.162 INFO:teuthology.orchestra.run.smithi033.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 838 kB 00:00 2024-03-27T02:04:27.237 INFO:teuthology.orchestra.run.smithi033.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-27T02:04:28.196 INFO:teuthology.orchestra.run.smithi033.stdout:(30/32): librbd1-19.0.0-2463.g4254c927.el9.x86_ 3.8 MB/s | 3.7 MB 00:00 2024-03-27T02:04:28.246 INFO:teuthology.orchestra.run.smithi033.stdout:(31/32): librados2-19.0.0-2463.g4254c927.el9.x8 3.1 MB/s | 3.4 MB 00:01 2024-03-27T02:04:33.154 INFO:teuthology.orchestra.run.smithi033.stdout:(32/32): ceph-common-19.0.0-2463.g4254c927.el9. 1.9 MB/s | 19 MB 00:09 2024-03-27T02:04:33.160 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:04:33.160 INFO:teuthology.orchestra.run.smithi033.stdout:Total 5.8 MB/s | 58 MB 00:10 2024-03-27T02:04:33.194 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:04:33.236 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:04:33.236 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:04:33.736 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:04:33.736 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:04:35.287 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:04:35.774 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-27T02:04:35.806 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-2463.g4254c927.el9.x86_64 2/34 2024-03-27T02:04:35.906 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-2463.g4254c927.el9.x86_64 2/34 2024-03-27T02:04:35.997 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-2463.g4254c927.el9.x86_64 3/34 2024-03-27T02:04:36.024 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-2463.g4254c927.el9.x86_64 4/34 2024-03-27T02:04:36.070 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-2463.g4254c927.el9.x86_64 4/34 2024-03-27T02:04:36.114 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-27T02:04:36.145 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-27T02:04:36.171 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-27T02:04:36.194 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T02:04:36.209 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-27T02:04:36.378 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T02:04:36.410 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-27T02:04:36.788 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-27T02:04:36.814 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-2463.g4254c927.el9.x86_64 11/34 2024-03-27T02:04:36.881 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-2463.g4254c927.el9.x86_64 11/34 2024-03-27T02:04:36.925 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-2463.g4254c927.el9. 12/34 2024-03-27T02:04:36.994 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-2463.g4254c927.el9.x86_64 13/34 2024-03-27T02:04:37.029 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-2463.g4254c927.el9.x86_64 14/34 2024-03-27T02:04:37.118 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-27T02:04:37.159 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-2463.g4254c927.el9.x86_6 16/34 2024-03-27T02:04:37.248 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2463.g4254c927.el9.x86_6 16/34 2024-03-27T02:04:37.283 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-27T02:04:37.394 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-27T02:04:37.426 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-27T02:04:37.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-27T02:04:37.499 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-27T02:04:37.528 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-27T02:04:37.559 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-27T02:04:37.596 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-27T02:04:38.163 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-27T02:04:38.191 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-2463.g4254c927.el9.x86_64 26/34 2024-03-27T02:04:38.245 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-2463.g4254c927.el9.x86_64 26/34 2024-03-27T02:04:38.316 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-2463.g4254c927.el9.x86_64 27/34 2024-03-27T02:04:38.368 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-2463.g4254c927.el9.x8 28/34 2024-03-27T02:04:40.834 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 29/34 2024-03-27T02:04:40.865 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 29/34 2024-03-27T02:04:41.597 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 29/34 2024-03-27T02:04:41.635 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-2463.g4254c927.el9.x86_64 30/34 2024-03-27T02:04:41.724 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-2463.g4254c927.el9.x86_64 30/34 2024-03-27T02:04:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-27T02:04:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-27T02:04:41.725 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:41.761 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-2463.g4254c927.el9.x86_64 31/34 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2463.g4254c927.el9.x86_64 31/34 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-03-27T02:04:52.050 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-03-27T02:04:52.051 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-03-27T02:04:52.051 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-03-27T02:04:52.051 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:53.100 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-2463.g4254c927.el9.x86_64 32/34 2024-03-27T02:04:53.124 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2463.g4254c927.el9.x86_64 32/34 2024-03-27T02:04:53.124 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:04:53.124 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-27T02:04:53.124 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T02:04:53.124 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-27T02:04:53.125 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:53.126 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T02:04:53.194 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-27T02:04:53.194 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T02:04:54.444 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T02:04:54.444 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-2463.g4254c927.el9.x86_64 1/34 2024-03-27T02:04:54.444 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 2/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-2463.g4254c927.el9.x86_64 3/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-2463.g4254c927.el9.x86_64 4/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-2463.g4254c927.el9.x86_64 5/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-2463.g4254c927.el9.x86_6 6/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-2463.g4254c927.el9.x86_64 7/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2463.g4254c927.el9. 8/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-2463.g4254c927.el9.x8 9/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-2463.g4254c927.el9.x86_64 10/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-2463.g4254c927.el9.x86_64 11/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-2463.g4254c927.el9.x86_64 12/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-2463.g4254c927.el9.x86_64 13/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-27T02:04:54.445 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-27T02:04:54.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-27T02:04:54.447 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-27T02:04:54.447 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-2463.g4254c927.el9.x86_64 31/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-27T02:04:54.448 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-2463.g4254c927.el9.x86_64 33/34 2024-03-27T02:04:55.441 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-27T02:04:55.441 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:55.441 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-03-27T02:04:55.441 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-27T02:04:55.442 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:55.444 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:04:55.777 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test 2024-03-27T02:04:56.371 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:00:36 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:04:56.528 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:56.529 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-2463.g4254c927.el9 ceph 87 M 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-2463.g4254c927.el9 ceph 173 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-27T02:04:56.530 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout:Install 11 Packages 2024-03-27T02:04:56.531 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 90 M 2024-03-27T02:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 819 M 2024-03-27T02:04:56.532 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:04:57.537 INFO:teuthology.orchestra.run.smithi033.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 683 kB/s | 217 kB 00:00 2024-03-27T02:04:57.604 INFO:teuthology.orchestra.run.smithi033.stdout:(2/11): libcephsqlite-19.0.0-2463.g4254c927.el9 451 kB/s | 173 kB 00:00 2024-03-27T02:04:57.637 INFO:teuthology.orchestra.run.smithi033.stdout:(3/11): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-03-27T02:04:57.662 INFO:teuthology.orchestra.run.smithi033.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-27T02:04:57.679 INFO:teuthology.orchestra.run.smithi033.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-27T02:04:57.721 INFO:teuthology.orchestra.run.smithi033.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.5 MB/s | 90 kB 00:00 2024-03-27T02:04:57.872 INFO:teuthology.orchestra.run.smithi033.stdout:(7/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 3.1 MB/s | 469 kB 00:00 2024-03-27T02:04:57.913 INFO:teuthology.orchestra.run.smithi033.stdout:(8/11): libubsan-11.4.1-3.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-27T02:04:58.256 INFO:teuthology.orchestra.run.smithi033.stdout:(9/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 384 kB/s | 122 kB 00:00 2024-03-27T02:04:58.573 INFO:teuthology.orchestra.run.smithi033.stdout:(10/11): jq-1.6-16.el9.x86_64.rpm 210 kB/s | 187 kB 00:00 2024-03-27T02:05:02.356 INFO:teuthology.orchestra.run.smithi033.stdout:(11/11): ceph-test-19.0.0-2463.g4254c927.el9.x8 17 MB/s | 87 MB 00:05 2024-03-27T02:05:02.360 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:05:02.360 INFO:teuthology.orchestra.run.smithi033.stdout:Total 15 MB/s | 90 MB 00:05 2024-03-27T02:05:02.403 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:05:02.440 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:05:02.440 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:05:03.104 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:05:03.104 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:05:04.153 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:05:04.250 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-27T02:05:04.283 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-27T02:05:04.307 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T02:05:04.383 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-27T02:05:04.415 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-27T02:05:04.473 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-27T02:05:04.521 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-27T02:05:04.552 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-27T02:05:04.592 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-27T02:05:04.637 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-27T02:05:04.662 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-2463.g4254c927.el9.x86_64 10/11 2024-03-27T02:05:16.816 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2463.g4254c927.el9.x86_64 10/11 2024-03-27T02:05:16.888 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-2463.g4254c927.el9.x86_64 11/11 2024-03-27T02:05:23.172 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-2:19.0.0-2463.g4254c927.el9.x86_64 11/11 2024-03-27T02:05:23.172 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-2463.g4254c927.el9.x86_64 1/11 2024-03-27T02:05:23.172 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-2463.g4254c927.el9.x86_64 2/11 2024-03-27T02:05:23.172 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-27T02:05:23.172 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-27T02:05:23.173 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2024-03-27T02:05:23.603 INFO:teuthology.orchestra.run.smithi033.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:23.604 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:05:23.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph 2024-03-27T02:05:24.434 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:04 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:05:24.595 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:05:24.597 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-27T02:05:24.597 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-2463.g4254c927.el9 ceph 6.5 k 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-2463.g4254c927.el9 ceph 2.2 M 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-2463.g4254c927.el9 ceph 1.6 M 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 249 k 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-2463.g4254c927.el9 ceph 5.0 M 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-2463.g4254c927.el9 ceph 71 M 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-27T02:05:24.598 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-27T02:05:24.599 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-27T02:05:24.600 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout:Install 36 Packages 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 84 M 2024-03-27T02:05:24.601 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 921 M 2024-03-27T02:05:24.602 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:05:25.177 INFO:teuthology.orchestra.run.smithi033.stdout:(1/36): ceph-19.0.0-2463.g4254c927.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-03-27T02:05:25.436 INFO:teuthology.orchestra.run.smithi033.stdout:(2/36): ceph-mds-19.0.0-2463.g4254c927.el9.x86_ 5.2 MB/s | 2.2 MB 00:00 2024-03-27T02:05:25.494 INFO:teuthology.orchestra.run.smithi033.stdout:(3/36): ceph-mgr-19.0.0-2463.g4254c927.el9.x86_ 3.2 MB/s | 1.6 MB 00:00 2024-03-27T02:05:25.603 INFO:teuthology.orchestra.run.smithi033.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2463.g4254 2.2 MB/s | 249 kB 00:00 2024-03-27T02:05:25.778 INFO:teuthology.orchestra.run.smithi033.stdout:(5/36): ceph-mon-19.0.0-2463.g4254c927.el9.x86_ 8.4 MB/s | 5.0 MB 00:00 2024-03-27T02:05:25.803 INFO:teuthology.orchestra.run.smithi033.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 933 kB/s | 187 kB 00:00 2024-03-27T02:05:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-27T02:05:25.845 INFO:teuthology.orchestra.run.smithi033.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 759 kB/s | 32 kB 00:00 2024-03-27T02:05:25.871 INFO:teuthology.orchestra.run.smithi033.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-27T02:05:25.896 INFO:teuthology.orchestra.run.smithi033.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 766 kB/s | 39 kB 00:00 2024-03-27T02:05:25.921 INFO:teuthology.orchestra.run.smithi033.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-27T02:05:25.980 INFO:teuthology.orchestra.run.smithi033.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-27T02:05:26.005 INFO:teuthology.orchestra.run.smithi033.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-27T02:05:26.030 INFO:teuthology.orchestra.run.smithi033.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-27T02:05:26.056 INFO:teuthology.orchestra.run.smithi033.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-27T02:05:26.148 INFO:teuthology.orchestra.run.smithi033.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-27T02:05:26.190 INFO:teuthology.orchestra.run.smithi033.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-03-27T02:05:26.240 INFO:teuthology.orchestra.run.smithi033.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-03-27T02:05:26.265 INFO:teuthology.orchestra.run.smithi033.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 578 kB/s | 43 kB 00:00 2024-03-27T02:05:26.282 INFO:teuthology.orchestra.run.smithi033.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-03-27T02:05:26.316 INFO:teuthology.orchestra.run.smithi033.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 321 kB/s | 11 kB 00:00 2024-03-27T02:05:26.350 INFO:teuthology.orchestra.run.smithi033.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 532 kB/s | 18 kB 00:00 2024-03-27T02:05:26.375 INFO:teuthology.orchestra.run.smithi033.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-27T02:05:26.400 INFO:teuthology.orchestra.run.smithi033.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-27T02:05:26.425 INFO:teuthology.orchestra.run.smithi033.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-03-27T02:05:26.451 INFO:teuthology.orchestra.run.smithi033.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-27T02:05:26.476 INFO:teuthology.orchestra.run.smithi033.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-03-27T02:05:26.501 INFO:teuthology.orchestra.run.smithi033.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-27T02:05:26.527 INFO:teuthology.orchestra.run.smithi033.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-27T02:05:26.543 INFO:teuthology.orchestra.run.smithi033.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-03-27T02:05:26.569 INFO:teuthology.orchestra.run.smithi033.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-27T02:05:26.586 INFO:teuthology.orchestra.run.smithi033.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 848 kB/s | 36 kB 00:00 2024-03-27T02:05:26.611 INFO:teuthology.orchestra.run.smithi033.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-27T02:05:26.636 INFO:teuthology.orchestra.run.smithi033.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 8.3 MB/s | 427 kB 00:00 2024-03-27T02:05:26.661 INFO:teuthology.orchestra.run.smithi033.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-03-27T02:05:28.870 INFO:teuthology.orchestra.run.smithi033.stdout:(36/36): ceph-osd-19.0.0-2463.g4254c927.el9.x86 21 MB/s | 71 MB 00:03 2024-03-27T02:05:28.874 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:05:28.874 INFO:teuthology.orchestra.run.smithi033.stdout:Total 20 MB/s | 84 MB 00:04 2024-03-27T02:05:28.899 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:05:28.945 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:05:28.946 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:05:29.712 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:05:29.713 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:05:31.336 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:05:31.397 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-27T02:05:31.431 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-27T02:05:31.458 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-27T02:05:31.532 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-27T02:05:31.574 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-27T02:05:31.607 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-27T02:05:31.703 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-27T02:05:31.779 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-27T02:05:31.826 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-27T02:05:31.901 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-27T02:05:31.953 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-27T02:05:31.990 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-27T02:05:32.065 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-27T02:05:32.128 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-27T02:05:32.330 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-27T02:05:32.377 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-27T02:05:32.416 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-27T02:05:32.459 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-27T02:05:32.488 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-27T02:05:32.567 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-27T02:05:32.599 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T02:05:32.632 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-27T02:05:32.673 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-27T02:05:32.708 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-27T02:05:32.750 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-27T02:05:32.843 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-27T02:05:32.881 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-27T02:05:32.955 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-27T02:05:33.097 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-27T02:05:33.230 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T02:05:33.436 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2463.g4254c927.el9. 31/36 2024-03-27T02:05:33.461 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-2463.g4254c927.el9.x86_64 32/36 2024-03-27T02:05:33.486 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2463.g4254c927.el9.x86_64 32/36 2024-03-27T02:05:33.486 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:05:33.487 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-27T02:05:33.487 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T02:05:33.487 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-27T02:05:33.487 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:45.102 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-2463.g4254c927.el9.x86_64 33/36 2024-03-27T02:05:45.128 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-2463.g4254c927.el9.x86_64 33/36 2024-03-27T02:05:45.128 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:05:45.129 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-27T02:05:45.129 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T02:05:45.129 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-27T02:05:45.129 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:45.694 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-2463.g4254c927.el9.x86_64 34/36 2024-03-27T02:05:45.717 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-2463.g4254c927.el9.x86_64 34/36 2024-03-27T02:05:45.717 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:05:45.717 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-27T02:05:45.717 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T02:05:45.718 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-27T02:05:45.718 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:45.961 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-2463.g4254c927.el9.x86_64 35/36 2024-03-27T02:05:45.984 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-2463.g4254c927.el9.x86_64 35/36 2024-03-27T02:05:45.985 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:05:45.985 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-27T02:05:45.985 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T02:05:45.985 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-27T02:05:45.985 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:46.054 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-2463.g4254c927.el9.x86_64 36/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-2:19.0.0-2463.g4254c927.el9.x86_64 36/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-2463.g4254c927.el9.x86_64 1/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-2463.g4254c927.el9.x86_64 2/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-2463.g4254c927.el9.x86_64 3/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-2463.g4254c927.el9.x86_64 4/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-2463.g4254c927.el9.x86_64 5/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2463.g4254c927.el9. 6/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-27T02:05:52.897 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-27T02:05:52.898 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-27T02:05:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-27T02:05:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-27T02:05:52.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-27T02:05:52.900 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-27T02:05:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-27T02:05:52.901 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-27T02:05:53.819 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-27T02:05:53.820 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-27T02:05:53.821 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:53.822 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:05:54.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base 2024-03-27T02:05:54.663 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:34 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:05:54.765 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:05:54.823 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:05:54.824 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:05:54.824 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:05:54.945 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install cephadm 2024-03-27T02:05:55.539 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:35 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:05:55.695 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 762 k 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:55.696 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 762 k 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 768 k 2024-03-27T02:05:55.697 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:05:56.137 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm-19.0.0-2463.g4254c927.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-03-27T02:05:56.137 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:05:56.138 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-03-27T02:05:56.138 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:05:56.143 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:05:56.143 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:05:56.149 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:05:56.150 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:05:56.225 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:05:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-2463.g4254c927.el9.noarch 1/1 2024-03-27T02:05:56.671 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-2463.g4254c927.el9.noarch 1/1 2024-03-27T02:05:57.064 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-2463.g4254c927.el9.noarch 1/1 2024-03-27T02:05:57.331 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-2463.g4254c927.el9.noarch 1/1 2024-03-27T02:05:57.332 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:57.332 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:05:57.332 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:05:57.332 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:57.332 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:05:57.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache 2024-03-27T02:05:58.112 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:38 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:05:58.270 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2463.g4254c927.el9 ceph 158 k 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 158 k 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 640 k 2024-03-27T02:05:58.271 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:05:58.501 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-19.0.0-2463.g4254c9 682 kB/s | 158 kB 00:00 2024-03-27T02:05:58.502 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:05:58.502 INFO:teuthology.orchestra.run.smithi033.stdout:Total 678 kB/s | 158 kB 00:00 2024-03-27T02:05:58.502 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:05:58.512 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:05:58.512 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:05:58.567 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:05:58.567 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:05:58.728 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:05:58.785 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2463.g4254c927. 1/1 2024-03-27T02:05:58.808 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2463.g4254c927. 1/1 2024-03-27T02:05:58.809 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:05:58.809 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-27T02:05:58.809 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2463.g4254c927. 1/1 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:05:59.810 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:00.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr 2024-03-27T02:06:00.657 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:40 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:00.756 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-mgr-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:00.811 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:00.812 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:00.812 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:00.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-dashboard 2024-03-27T02:06:01.453 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:41 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:01.607 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 3.5 M 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:01.608 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 34 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 16 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-27T02:06:01.609 INFO:teuthology.orchestra.run.smithi033.stdout:Install 11 Packages 2024-03-27T02:06:01.610 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:01.610 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 7.6 M 2024-03-27T02:06:01.610 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 100 M 2024-03-27T02:06:01.610 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:01.988 INFO:teuthology.orchestra.run.smithi033.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2463.g42 129 kB/s | 34 kB 00:00 2024-03-27T02:06:02.013 INFO:teuthology.orchestra.run.smithi033.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2463.g425 57 kB/s | 16 kB 00:00 2024-03-27T02:06:02.147 INFO:teuthology.orchestra.run.smithi033.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-27T02:06:02.322 INFO:teuthology.orchestra.run.smithi033.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-27T02:06:02.481 INFO:teuthology.orchestra.run.smithi033.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-27T02:06:02.506 INFO:teuthology.orchestra.run.smithi033.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-27T02:06:02.631 INFO:teuthology.orchestra.run.smithi033.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-27T02:06:02.690 INFO:teuthology.orchestra.run.smithi033.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 535 kB/s | 31 kB 00:00 2024-03-27T02:06:02.732 INFO:teuthology.orchestra.run.smithi033.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 8.1 MB/s | 2.0 MB 00:00 2024-03-27T02:06:02.757 INFO:teuthology.orchestra.run.smithi033.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-27T02:06:03.365 INFO:teuthology.orchestra.run.smithi033.stdout:(11/11): ceph-mgr-dashboard-19.0.0-2463.g4254c9 2.2 MB/s | 3.5 MB 00:01 2024-03-27T02:06:03.370 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:03.370 INFO:teuthology.orchestra.run.smithi033.stdout:Total 4.3 MB/s | 7.6 MB 00:01 2024-03-27T02:06:03.390 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:03.409 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:03.409 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:03.595 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:03.595 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:04.089 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:04.173 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-27T02:06:04.210 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-27T02:06:04.302 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-27T02:06:04.441 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-27T02:06:04.497 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-27T02:06:04.583 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-27T02:06:04.633 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-27T02:06:04.660 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-27T02:06:04.693 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2463.g4254c927.el9 9/11 2024-03-27T02:06:05.800 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2463.g4254c927.el 10/11 2024-03-27T02:06:05.835 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2463.g4254c927.el9.noa 11/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2463.g4254c927.el9.noa 11/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2463.g4254c927.el 1/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2463.g4254c927.el9.noa 2/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2463.g4254c927.el9 3/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-27T02:06:07.052 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-27T02:06:07.053 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-27T02:06:07.053 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-27T02:06:07.053 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-27T02:06:07.053 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-27T02:06:07.053 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:07.398 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:07.399 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:07.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-27T02:06:08.204 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:48 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:08.364 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 7.4 M 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-27T02:06:08.365 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2024-03-27T02:06:08.366 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:08.367 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 37 M 2024-03-27T02:06:08.367 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 209 M 2024-03-27T02:06:08.367 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:08.831 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 198 kB/s | 32 kB 00:00 2024-03-27T02:06:08.872 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 418 kB/s | 17 kB 00:00 2024-03-27T02:06:08.915 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-27T02:06:08.973 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-03-27T02:06:09.157 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-27T02:06:09.224 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-27T02:06:09.249 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-27T02:06:09.466 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-27T02:06:09.567 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 8.3 MB/s | 7.4 MB 00:00 2024-03-27T02:06:09.642 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-27T02:06:09.725 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 755 kB/s | 195 kB 00:00 2024-03-27T02:06:10.150 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 420 kB/s | 245 kB 00:00 2024-03-27T02:06:10.152 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:10.153 INFO:teuthology.orchestra.run.smithi033.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-27T02:06:10.196 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:10.222 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:10.222 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:10.594 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:10.595 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:11.344 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:11.407 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-27T02:06:11.438 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-27T02:06:11.536 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-27T02:06:11.583 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-27T02:06:11.768 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T02:06:11.798 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T02:06:11.877 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-27T02:06:12.571 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-27T02:06:12.690 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-27T02:06:14.023 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-27T02:06:15.101 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-27T02:06:15.131 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2463.g4254c 12/12 2024-03-27T02:06:17.683 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2463.g4254c 12/12 2024-03-27T02:06:17.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2463.g4254c 1/12 2024-03-27T02:06:17.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-27T02:06:17.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-27T02:06:17.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-27T02:06:17.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-27T02:06:17.685 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-27T02:06:18.115 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-27T02:06:18.116 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-27T02:06:18.117 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-27T02:06:18.117 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-27T02:06:18.117 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-27T02:06:18.117 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:18.117 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:18.507 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-rook 2024-03-27T02:06:19.082 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:59 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:19.239 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:19.240 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 49 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-27T02:06:19.241 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2024-03-27T02:06:19.242 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:19.243 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.0 M 2024-03-27T02:06:19.243 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2024-03-27T02:06:19.243 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:19.497 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 135 kB/s | 24 kB 00:00 2024-03-27T02:06:19.522 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 18 kB 00:00 2024-03-27T02:06:19.581 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-27T02:06:19.606 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-27T02:06:19.640 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-27T02:06:19.665 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 736 kB/s | 43 kB 00:00 2024-03-27T02:06:19.924 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): ceph-mgr-rook-19.0.0-2463.g4254c927.el9 81 kB/s | 49 kB 00:00 2024-03-27T02:06:20.016 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-03-27T02:06:20.066 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 75 kB/s | 32 kB 00:00 2024-03-27T02:06:20.241 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 338 kB/s | 59 kB 00:00 2024-03-27T02:06:20.392 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 598 kB/s | 90 kB 00:00 2024-03-27T02:06:20.517 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 342 kB/s | 203 kB 00:00 2024-03-27T02:06:20.567 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-27T02:06:20.572 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:20.573 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-27T02:06:20.573 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:20.589 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:20.589 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:20.694 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:20.694 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:21.339 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:21.504 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-27T02:06:21.553 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-27T02:06:21.618 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-27T02:06:21.653 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-27T02:06:21.695 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-27T02:06:21.791 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-27T02:06:22.030 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-27T02:06:22.080 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-27T02:06:22.846 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-27T02:06:22.893 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-27T02:06:22.926 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-27T02:06:22.975 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-27T02:06:23.004 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-2463.g4254c927.el9.noarch 13/13 2024-03-27T02:06:23.747 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2463.g4254c927.el9.noarch 13/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2463.g4254c927.el9.noarch 1/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-27T02:06:23.748 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-27T02:06:23.749 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-27T02:06:24.113 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-27T02:06:24.114 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:24.115 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:24.377 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-cephadm 2024-03-27T02:06:24.947 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:04 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:25.107 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:25.107 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 141 k 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:25.108 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.8 M 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 33 M 2024-03-27T02:06:25.109 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:25.619 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 753 kB/s | 248 kB 00:00 2024-03-27T02:06:25.844 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2024-03-27T02:06:25.877 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2463.g4254c927.e 240 kB/s | 141 kB 00:00 2024-03-27T02:06:26.019 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-27T02:06:26.044 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 288 kB/s | 58 kB 00:00 2024-03-27T02:06:26.070 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 304 kB/s | 51 kB 00:00 2024-03-27T02:06:26.074 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:26.074 INFO:teuthology.orchestra.run.smithi033.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-27T02:06:26.086 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:26.097 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:26.097 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:26.190 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:26.190 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:26.493 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:26.670 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-27T02:06:26.723 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-27T02:06:27.280 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-27T02:06:27.362 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-27T02:06:27.432 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-27T02:06:27.460 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2463.g4254c927.el9.noarch 6/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2463.g4254c927.el9.noarch 6/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2463.g4254c927.el9.noarch 1/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-27T02:06:28.202 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-27T02:06:28.518 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-27T02:06:28.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:28.518 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:28.519 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:28.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse 2024-03-27T02:06:29.362 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:09 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:29.521 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-2463.g4254c927.el9 ceph 870 k 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:29.522 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-27T02:06:29.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 949 k 2024-03-27T02:06:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 4.0 M 2024-03-27T02:06:29.523 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:29.656 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 589 kB/s | 79 kB 00:00 2024-03-27T02:06:30.407 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-fuse-19.0.0-2463.g4254c927.el9.x86_ 985 kB/s | 870 kB 00:00 2024-03-27T02:06:30.408 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:30.409 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.0 MB/s | 949 kB 00:00 2024-03-27T02:06:30.409 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:30.436 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:30.436 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:30.517 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:30.517 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:30.709 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:30.862 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-27T02:06:30.943 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 2/2 2024-03-27T02:06:31.634 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 2/2 2024-03-27T02:06:31.635 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 1/2 2024-03-27T02:06:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-27T02:06:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:31.960 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-27T02:06:31.961 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:31.961 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:32.224 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-volume 2024-03-27T02:06:32.799 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:12 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:32.956 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-2463.g4254c927.el9 ceph-noarch 264 k 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-27T02:06:32.957 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 332 k 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.5 M 2024-03-27T02:06:32.958 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:33.086 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 539 kB/s | 69 kB 00:00 2024-03-27T02:06:33.619 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-volume-19.0.0-2463.g4254c927.el9.no 399 kB/s | 264 kB 00:00 2024-03-27T02:06:33.620 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:33.621 INFO:teuthology.orchestra.run.smithi033.stdout:Total 501 kB/s | 332 kB 00:00 2024-03-27T02:06:33.621 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:33.631 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:33.631 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:33.680 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:33.680 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:33.909 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:34.082 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-27T02:06:34.114 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-2463.g4254c927.el9.noarch 2/2 2024-03-27T02:06:34.130 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-2463.g4254c927.el9.noarch 2/2 2024-03-27T02:06:34.131 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:06:34.131 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-27T02:06:34.131 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:34.874 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-2463.g4254c927.el9.noarch 1/2 2024-03-27T02:06:35.228 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-27T02:06:35.228 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:35.228 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-2463.g4254c927.el9.noarch 2024-03-27T02:06:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-27T02:06:35.229 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:35.229 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:35.511 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel 2024-03-27T02:06:36.092 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:16 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:36.249 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:36.250 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-27T02:06:36.250 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:36.250 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:36.250 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-2463.g4254c927.el9 ceph 125 k 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 125 k 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 445 k 2024-03-27T02:06:36.251 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:36.567 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-19.0.0-2463.g4254c927.el9.x86_64 395 kB/s | 125 kB 00:00 2024-03-27T02:06:36.567 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:36.568 INFO:teuthology.orchestra.run.smithi033.stdout:Total 393 kB/s | 125 kB 00:00 2024-03-27T02:06:36.568 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:36.574 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:36.574 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:36.634 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:36.634 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:36.789 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:36.902 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:37.377 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:37.701 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:37.701 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:37.701 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:37.701 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:06:37.702 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:37.702 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:37.920 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2 2024-03-27T02:06:38.503 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:18 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:38.605 INFO:teuthology.orchestra.run.smithi033.stdout:Package libcephfs2-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:38.660 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:38.661 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:38.661 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:38.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs-devel 2024-03-27T02:06:39.297 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:19 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:39.454 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-2463.g4254c927.el9 ceph 32 k 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:39.455 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:06:39.456 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:39.456 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 32 k 2024-03-27T02:06:39.456 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 142 k 2024-03-27T02:06:39.456 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:39.620 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs-devel-19.0.0-2463.g4254c927.el9.x86_6 193 kB/s | 32 kB 00:00 2024-03-27T02:06:39.620 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:39.620 INFO:teuthology.orchestra.run.smithi033.stdout:Total 192 kB/s | 32 kB 00:00 2024-03-27T02:06:39.621 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:39.624 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:39.624 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:39.657 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:39.657 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:39.745 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:39.864 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:40.219 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:40.494 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:40.668 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2 2024-03-27T02:06:41.243 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:21 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:41.344 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:41.401 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:41.401 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:41.474 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1 2024-03-27T02:06:42.047 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:22 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:42.149 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:42.204 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:42.205 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:42.205 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:42.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados 2024-03-27T02:06:42.832 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:22 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:42.934 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rados-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:42.990 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:42.991 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:42.992 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:43.057 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw 2024-03-27T02:06:43.629 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:23 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:43.730 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rgw-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:43.785 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:43.786 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:43.786 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:43.850 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs 2024-03-27T02:06:44.423 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:24 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:44.528 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-cephfs-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:44.583 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:44.585 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:44.585 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:44.647 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd 2024-03-27T02:06:45.218 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:25 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:45.317 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rbd-2:19.0.0-2463.g4254c927.el9.x86_64 is already installed. 2024-03-27T02:06:45.372 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:45.373 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-27T02:06:45.373 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:45.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse 2024-03-27T02:06:46.011 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:26 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:46.167 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-2463.g4254c927.el9 ceph 90 k 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:46.168 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:46.169 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:06:46.169 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:46.169 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 90 k 2024-03-27T02:06:46.169 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 313 k 2024-03-27T02:06:46.169 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:46.372 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-19.0.0-2463.g4254c927.el9.x86_64.rpm 442 kB/s | 90 kB 00:00 2024-03-27T02:06:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:Total 439 kB/s | 90 kB 00:00 2024-03-27T02:06:46.373 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:46.383 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:46.383 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:46.442 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:46.442 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:46.587 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:46.699 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:47.060 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:47.350 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:47.596 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror 2024-03-27T02:06:48.167 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:28 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:48.324 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:48.324 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:48.324 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-27T02:06:48.324 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:48.324 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-2463.g4254c927.el9 ceph 3.4 M 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:48.325 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.4 M 2024-03-27T02:06:48.326 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 29 M 2024-03-27T02:06:48.326 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:48.717 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-19.0.0-2463.g4254c927.el9.x86_64.rpm 8.7 MB/s | 3.4 MB 00:00 2024-03-27T02:06:48.718 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:48.718 INFO:teuthology.orchestra.run.smithi033.stdout:Total 8.7 MB/s | 3.4 MB 00:00 2024-03-27T02:06:48.718 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:48.730 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:48.730 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:48.808 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:48.809 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:49.401 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:49.468 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:49.492 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:49.492 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-27T02:06:49.492 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-27T02:06:49.493 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T02:06:49.493 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-27T02:06:49.493 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:50.709 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:50.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd 2024-03-27T02:06:51.509 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:31 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:06:51.665 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:06:51.666 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:51.666 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-2463.g4254c927.el9 ceph 175 k 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 175 k 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 673 k 2024-03-27T02:06:51.667 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:06:51.987 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-19.0.0-2463.g4254c927.el9.x86_64.rpm 547 kB/s | 175 kB 00:00 2024-03-27T02:06:51.988 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:06:51.988 INFO:teuthology.orchestra.run.smithi033.stdout:Total 545 kB/s | 175 kB 00:00 2024-03-27T02:06:51.988 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:06:51.999 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:06:51.999 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:06:52.059 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:06:52.059 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:06:52.227 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:06:52.323 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:52.740 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:53.032 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-2463.g4254c927.el9.x86_64 1/1 2024-03-27T02:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:53.033 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:06:53.267 DEBUG:teuthology.parallel:result is None 2024-03-27T02:06:53.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4254c92712efa5f9025679283afe440af32d0589 2024-03-27T02:06:53.516 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-27T02:06:53.537 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-2463.g4254c927.el9 2024-03-27T02:06:53.538 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2463.g4254c927.el9 2024-03-27T02:06:53.538 INFO:teuthology.task.install:The correct ceph version 19.0.0-2463.g4254c927 is installed. 2024-03-27T02:06:53.540 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-27T02:06:53.540 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:53.540 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-27T02:06:53.606 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-27T02:06:53.606 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:53.606 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-03-27T02:06:53.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-27T02:06:53.740 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-27T02:06:53.740 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:53.740 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-27T02:06:53.807 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-27T02:06:53.896 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-27T02:06:53.896 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:53.896 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-03-27T02:06:53.923 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-27T02:06:53.990 INFO:teuthology.run_tasks:Running task ceph... 2024-03-27T02:06:54.079 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-27T02:06:54.079 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2024-03-27T02:06:54.105 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-27T02:06:54.105 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-27T02:06:54.173 INFO:tasks.ceph:Creating extra log directories... 2024-03-27T02:06:54.173 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-27T02:06:54.249 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-27T02:06:54.250 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-27T02:06:54.250 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043', 'branch': 'ci-yingxin-crimson-load-obc-2', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7625043', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-2'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4254c92712efa5f9025679283afe440af32d0589'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4254c92712efa5f9025679283afe440af32d0589'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-2', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4254c92712efa5f9025679283afe440af32d0589'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '4254c92712efa5f9025679283afe440af32d0589', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-2', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_4254c92712efa5f9025679283afe440af32d0589/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '4254c92712efa5f9025679283afe440af32d0589', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHGOCRXDIc5f5ZrdgQ43O5wUWTbQPI4Eq8i7O2NCIzQJPCON69xoODrNGjrxYrIAZcxP7Wj9o9smcQmSDZoLT7Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4254c92712efa5f9025679283afe440af32d0589', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-27_01:32:53', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-27T02:06:54.250 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-27T02:06:54.297 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2024-03-27T02:06:54.365 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:54.365 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-27T02:06:54.420 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-27T02:06:54.420 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-27 02:06:49.728149097 +0000 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-27 02:03:27.448553409 +0000 2024-03-27T02:06:54.475 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-27 02:03:27.448553409 +0000 2024-03-27T02:06:54.476 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-27 02:03:27.448553409 +0000 2024-03-27T02:06:54.476 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-27T02:06:54.541 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-27T02:06:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-27T02:06:54.542 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000202885 s, 2.5 MB/s 2024-03-27T02:06:54.543 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-27T02:06:54.599 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-27 02:06:49.728149097 +0000 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-27 02:03:27.797547495 +0000 2024-03-27T02:06:54.654 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-27 02:03:27.797547495 +0000 2024-03-27T02:06:54.655 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-27 02:03:27.797547495 +0000 2024-03-27T02:06:54.655 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-27T02:06:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-27T02:06:54.720 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-27T02:06:54.721 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00017145 s, 3.0 MB/s 2024-03-27T02:06:54.722 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-27T02:06:54.777 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-27 02:06:49.728149097 +0000 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-27 02:03:28.131541835 +0000 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-27 02:03:28.131541835 +0000 2024-03-27T02:06:54.833 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-27 02:03:28.131541835 +0000 2024-03-27T02:06:54.834 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-27T02:06:54.898 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-27T02:06:54.899 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-27T02:06:54.899 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000193552 s, 2.6 MB/s 2024-03-27T02:06:54.900 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-27T02:06:54.955 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-27T02:06:55.009 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-27 02:06:49.728149097 +0000 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-27 02:03:28.469536108 +0000 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-27 02:03:28.469536108 +0000 2024-03-27T02:06:55.010 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-27 02:03:28.469536108 +0000 2024-03-27T02:06:55.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-27T02:06:55.074 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-27T02:06:55.075 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-27T02:06:55.075 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000212651 s, 2.4 MB/s 2024-03-27T02:06:55.076 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-27T02:06:55.131 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T02:06:55.131 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-27T02:06:55.132 INFO:tasks.ceph:Generating config... 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-27T02:06:55.133 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] log to file = False 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-27T02:06:55.134 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-27T02:06:55.135 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-27T02:06:55.135 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] auth client required = none 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] auth cluster required = none 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] auth service required = none 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] auth supported = none 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug asok = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug auth = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug client = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-03-27T02:06:55.135 INFO:tasks.ceph:[global] debug context = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug crush = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug filer = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug journal = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-03-27T02:06:55.136 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug mon = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug monc = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug ms = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-03-27T02:06:55.137 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug osd = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rados = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug refs = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-03-27T02:06:55.138 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] debug striper = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] debug timer = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] debug tp = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-27T02:06:55.139 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-27T02:06:55.139 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-27T02:06:55.139 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-03-27T02:06:55.139 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-03-27T02:06:55.140 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-27T02:06:55.140 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-27T02:06:55.140 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-27T02:06:55.140 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-27T02:06:55.140 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-27T02:06:55.140 INFO:tasks.ceph:Setting up mon.a... 2024-03-27T02:06:55.140 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-27T02:06:55.228 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2024-03-27T02:06:55.231 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-27T02:06:55.283 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T02:06:55.309 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.33')] 2024-03-27T02:06:55.310 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-27T02:06:55.311 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:55.311 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T02:06:55.364 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:06:55.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:06:55.467+0000 7f4b43451f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:06:55.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:06:55.467+0000 7f4b43451f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:06:55.455 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:06:55.467+0000 7f4b43451f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = quincy 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:fsid 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2024-03-27T02:06:55.468575+0000 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:created 2024-03-27T02:06:55.468575+0000 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 17 (quincy) 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2024-03-27T02:06:55.456 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-27T02:06:55.457 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-27T02:06:55.472 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 80632d94-d178-4e29-aef4-920a112be4f8... 2024-03-27T02:06:55.473 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = none 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.562 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2024-03-27T02:06:55.563 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.564 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = 172.21.15.33 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: auth client required = none 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: auth cluster required = none 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: auth service required = none 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug asok = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug bdev = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug buffer = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug civetweb = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug client = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug compressor = 0/0 2024-03-27T02:06:55.565 INFO:teuthology.orchestra.run.smithi033.stdout: debug context = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug crush = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug crypto = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug dpdk = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug eventtrace = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug filer = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug filestore = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug finisher = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug fuse = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug heartbeatmap = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug javaclient = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug journal = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug journaler = 0/0 2024-03-27T02:06:55.566 INFO:teuthology.orchestra.run.smithi033.stdout: debug kstore = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug leveldb = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug lockdep = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds balancer = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds locker = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds log = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds log expire = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds migrator = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug memdb = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgrc = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 0/0 2024-03-27T02:06:55.567 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug objclass = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug objectcacher = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug objecter = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug optracker = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug perfcounter = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rados = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd mirror = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd replay = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug refs = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw lifecycle = 0/0 2024-03-27T02:06:55.568 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw notification = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw sync = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug striper = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug throttle = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug timer = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: debug tp = 0/0 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms client mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms cluster mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon client mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon cluster mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon service mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: ms service mode = crc 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:06:55.569 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2024-03-27T02:06:55.570 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2024-03-27T02:06:55.571 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression algorithm = snappy 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression mode = aggressive 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore rocksdb cf = False 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: crimson alien thread cpu cores = 6-7 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: crimson osd obc lru size = 10 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 20 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 20 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: debug filestore = 0/0 2024-03-27T02:06:55.572 INFO:teuthology.orchestra.run.smithi033.stdout: debug journal = 0/0 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 0/0 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: err to stderr = True 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: log flush on exit = True 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: log to file = False 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: log to stderr = True 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.573 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 0/0 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 0/0 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2024-03-27T02:06:55.574 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.575 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default crimson = True 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout: crimson seastar cpu cores = 0-2 2024-03-27T02:06:55.576 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2024-03-27T02:06:55.577 INFO:teuthology.orchestra.run.smithi033.stdout: crimson seastar cpu cores = 3-5 2024-03-27T02:06:55.577 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2024-03-27T02:06:55.577 INFO:teuthology.orchestra.run.smithi033.stdout: crimson seastar cpu cores = 0-2 2024-03-27T02:06:55.577 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2024-03-27T02:06:55.577 INFO:teuthology.orchestra.run.smithi033.stdout: crimson seastar cpu cores = 3-5 2024-03-27T02:06:55.578 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-27T02:06:55.578 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-27T02:06:55.643 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-27T02:06:55.643 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:55.643 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-27T02:06:55.658 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:55.658 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-27T02:06:55.712 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:06:55.712 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:55.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-27T02:06:55.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-27T02:06:55.791 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:06:55.791 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:06:55.846 INFO:tasks.ceph:Setting up mon nodes... 2024-03-27T02:06:55.846 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-27T02:06:55.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T02:06:55.948 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-27T02:06:55.951 INFO:tasks.ceph:Setting up mds nodes... 2024-03-27T02:06:55.951 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-27T02:06:55.951 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-27T02:06:56.001 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-27T02:06:56.017 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-27T02:06:56.017 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-27T02:06:56.017 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-27T02:06:56.044 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T02:06:56.044 INFO:tasks.ceph:role: osd.0 2024-03-27T02:06:56.044 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:06:56.044 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-27T02:06:56.118 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T02:06:56.118 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T02:06:56.119 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T02:06:56.615 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-27T02:06:56.617 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-27T02:06:56.617 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-27T02:06:56.738 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-27T02:06:56.773 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-27T02:06:56.841 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T02:06:56.842 INFO:tasks.ceph:role: osd.1 2024-03-27T02:06:56.842 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:06:56.842 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-27T02:06:56.917 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T02:06:56.918 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T02:06:57.381 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-27T02:06:57.383 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-27T02:06:57.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-27T02:06:57.482 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-27T02:06:57.514 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-27T02:06:57.583 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-27T02:06:57.583 INFO:tasks.ceph:role: osd.2 2024-03-27T02:06:57.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:06:57.583 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-27T02:06:57.659 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-27T02:06:57.660 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-27T02:06:57.660 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-27T02:06:58.193 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-27T02:06:58.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-27T02:06:58.195 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-27T02:06:58.310 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-27T02:06:58.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:06:58.668 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:06:58.669 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:06:58,683 seastar - Reactor backend: linux-aio 2024-03-27T02:06:58.680 INFO:teuthology.orchestra.run.smithi033.stderr:==29120==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:06:58.823 INFO:teuthology.orchestra.run.smithi033.stderr:DEBUG 2024-03-27 02:06:58,837 [shard 0:main] seastar - reactor::drain 2024-03-27T02:06:59.380 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:06:59.380 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:06:59.380 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:06:59.380 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:06:59.380 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:06:59.381 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:06:59.438 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:06:59.442 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:06:59,455 seastar - Reactor backend: linux-aio 2024-03-27T02:06:59.451 INFO:teuthology.orchestra.run.smithi033.stderr:==29118==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:06:59.516 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb974779 0xb847240 0xb8487bc 0xb8489d2 0xb848b28 0xb848ff1 0x54daf 0x2e9e 0x4a09d 0x35b7c 0x3629e 0x5d8bcf4 0x5d8bef0 0x5d8c0ed 0x5d8c3c3 0x5ddac03 0x5ddae09 0x5d598f0 0x5d59e24 0x5b9ee07 0x5bb9ecb 0x6e94484 0x6ea5c57 0x6ea70f4 0x6ea7179 0x6ea727b 0x6ea7373 0x6ea7580 0x6ea7766 0x6ea7ab5 0x6ea7e59 0x6ea8089 0x6ea82f3 0x6ea85c0 0x6ea87de 0x6ea8896 0x6ea895c 0x6ea8a3d 0x5a0540e 0x5a05518 0x5a055d4 0x5a05690 0x5a0574c 0x5a05899 0x5a05a24 0x5a05de5 0x5a22862 0xb859ec5 0xb87415f 0xb91515d 0xb9167ab 0xb5f2f13 0xb5f388f 0x362558a 0x3feaf 0x3ff5f 0x3411444 2024-03-27T02:06:59.516 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:04.145 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:04.146 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:07:04.168 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-27T02:07:04.195 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:07:04.484 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:07:04.485 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:07:04,499 seastar - Reactor backend: linux-aio 2024-03-27T02:07:04.490 INFO:teuthology.orchestra.run.smithi033.stderr:==29302==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:07:04.598 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb974779 0xb847240 0xb8487bc 0xb8489d2 0xb848b28 0xb848ff1 0x54daf 0x4a0ae 0x35b7c 0x3689e 0x363b5de 0x5d7e5b3 0x5d7eccb 0x5d7edf1 0x5ddad74 0x5d598f0 0x5d59e24 0x5b9ee07 0x5bb9ecb 0x6e94484 0x6ea5c57 0x6ea70f4 0x6ea7179 0x6ea727b 0x6ea7373 0x6ea7580 0x6ea7766 0x6ea7ab5 0x6ea7e59 0x6ea8089 0x6ea82f3 0x6ea85c0 0x6ea87de 0x6ea8896 0x6ea895c 0x6ea8a3d 0x5a0540e 0x5a05518 0x5a055d4 0x5a05690 0x5a0574c 0x5a05899 0x5a05a24 0x5a05de5 0x5a22862 0xb859ec5 0xb87415f 0xb91515d 0xb9167ab 0xb5f2f13 0xb5f388f 0x3784266 0x3784dbf 0x3624d7a 0x3feaf 0x3ff5f 0x3411444 2024-03-27T02:07:04.599 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2024-03-27T02:07:04.635 INFO:teuthology.orchestra.run.smithi033.stderr:DEBUG 2024-03-27 02:07:04,649 [shard 0:main] seastar - reactor::drain 2024-03-27T02:07:05.181 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:07:05.182 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:07:05.182 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:07:05.182 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:05.182 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:05.241 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:07:05.245 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:07:05,258 seastar - Reactor backend: linux-aio 2024-03-27T02:07:05.254 INFO:teuthology.orchestra.run.smithi033.stderr:==29300==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:07:05.319 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb974779 0xb847240 0xb8487bc 0xb8489d2 0xb848b28 0xb848ff1 0x54daf 0x26513 0xef7ec 0x4a0a5 0x35b7c 0x3629e 0x5d8bcf4 0x5d8bed6 0x5d8c0ed 0x5d8c3c3 0x5ddac03 0x5ddae09 0x5d598f0 0x5d59aca 0x5d5a056 0x5b9ee07 0x5bb9ecb 0x6e94484 0x6ea5c57 0x6ea70f4 0x6ea7179 0x6ea727b 0x6ea7373 0x6ea7580 0x6ea7766 0x6ea7ab5 0x6ea7e59 0x6ea8089 0x6ea82f3 0x6ea85c0 0x6ea87de 0x6ea8896 0x6ea895c 0x6ea8a3d 0x5a0540e 0x5a05518 0x5a055d4 0x5a05690 0x5a0574c 0x5a05899 0x5a05a24 0x5a05de5 0x5a22862 0xb859ec5 0xb87415f 0xb91515d 0xb9167ab 0xb5f2f13 0xb5f388f 0x362558a 0x3feaf 0x3ff5f 0x3411444 2024-03-27T02:07:05.320 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:09.942 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:07:09.964 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-27T02:07:09.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:07:10.282 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:07:10.283 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:07:10,296 seastar - Reactor backend: linux-aio 2024-03-27T02:07:10.288 INFO:teuthology.orchestra.run.smithi033.stderr:==29484==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:07:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb974779 0xb847240 0xb8487bc 0xb8489d2 0xb848b28 0xb848ff1 0x54daf 0x34694a7 0x34695d7 0x5d5a704 0x5b9ee07 0x5bb9ecb 0x6e94484 0x6ea5c57 0x6ea70f4 0x6ea7179 0x6ea727b 0x6ea7373 0x6ea7580 0x6ea7766 0x6ea7ab5 0x6ea7e59 0x6ea8089 0x6ea82f3 0x6ea85c0 0x6ea87de 0x6ea8896 0x6ea895c 0x6ea8a3d 0x5a0540e 0x5a05518 0x5a055d4 0x5a05690 0x5a0574c 0x5a05899 0x5a05a24 0x5a05de5 0x5a22862 0xb859ec5 0xb87415f 0xb91515d 0xb9167ab 0xb5f2f13 0xb5f388f 0x3784266 0x3784dbf 0x3624d7a 0x3feaf 0x3ff5f 0x3411444 2024-03-27T02:07:10.396 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2024-03-27T02:07:10.431 INFO:teuthology.orchestra.run.smithi033.stderr:DEBUG 2024-03-27 02:07:10,444 [shard 0:main] seastar - reactor::drain 2024-03-27T02:07:10.996 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:10.997 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:07:10.997 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:07:10.997 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:07:10.997 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:10.997 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:11.052 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-27T02:07:11.056 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2024-03-27 02:07:11,069 seastar - Reactor backend: linux-aio 2024-03-27T02:07:11.065 INFO:teuthology.orchestra.run.smithi033.stderr:==29482==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-27T02:07:11.130 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb974779 0xb847240 0xb8487bc 0xb8489d2 0xb848b28 0xb848ff1 0x54daf 0x174af 0x2eab 0x4a09d 0x35b7c 0x3629e 0x5ddac6b 0x5ddae09 0x5d598f0 0x5d59e24 0x5b9ee07 0x5bb9ecb 0x6e94484 0x6ea5c57 0x6ea70f4 0x6ea7179 0x6ea727b 0x6ea7373 0x6ea7580 0x6ea7766 0x6ea7ab5 0x6ea7e59 0x6ea8089 0x6ea82f3 0x6ea85c0 0x6ea87de 0x6ea8896 0x6ea895c 0x6ea8a3d 0x5a0540e 0x5a05518 0x5a055d4 0x5a05690 0x5a0574c 0x5a05899 0x5a05a24 0x5a05de5 0x5a22862 0xb859ec5 0xb87415f 0xb91515d 0xb9167ab 0xb5f2f13 0xb5f388f 0x362558a 0x3feaf 0x3ff5f 0x3411444 2024-03-27T02:07:11.131 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2024-03-27T02:07:15.737 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:15.737 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2024-03-27T02:07:15.738 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2024-03-27T02:07:15.738 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2024-03-27T02:07:15.738 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2024-03-27T02:07:15.738 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:15.738 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 80632d94-d178-4e29-aef4-920a112be4f8 2024-03-27T02:07:15.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-27T02:07:15.787 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-27T02:07:15.787 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:15.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-27T02:07:15.854 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:15.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-27T02:07:15.920 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:15.921 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-27T02:07:15.986 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:15.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-27T02:07:16.053 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:16.053 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-27T02:07:16.108 INFO:tasks.ceph:Adding keys to all mons... 2024-03-27T02:07:16.108 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQA/fwNm9PBVORAABB85AE8YH2a1I5VvJDHQHA== 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBDfwNm0FgzJRAA9a5qDRnNnADn2BDmi6ig/Q== 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBJfwNmhPWYGRAAtG481wmUr9PTqjUOmJ1SrQ== 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBPfwNmutQXDhAA51O61TEahDi2QCcbit3A3g== 2024-03-27T02:07:16.173 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-03-27T02:07:16.174 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBAfwNmfrXlABAAfvAJP+goBA8frw4tIsDSkg== 2024-03-27T02:07:16.175 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-27T02:07:16.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T02:07:16.337 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T02:07:16.400 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-27T02:07:16.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-27T02:07:16.525 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-27T02:07:16.525 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-27T02:07:16.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-27T02:07:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:16.673+0000 7f7117c76a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:16.673+0000 7f7117c76a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.676 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-27T02:07:16.703 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-27T02:07:16.758 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-27T02:07:16.758 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-27T02:07:16.758 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-27T02:07:16.801 INFO:tasks.ceph.mon.a:Started 2024-03-27T02:07:16.801 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-27T02:07:16.802 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-27T02:07:16.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-27T02:07:16.804 INFO:tasks.ceph.mgr.x:Started 2024-03-27T02:07:16.804 INFO:tasks.ceph:Setting crush tunables to default 2024-03-27T02:07:16.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2024-03-27T02:07:16.887 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-03-27T02:07:16.900+0000 7f3afaa33a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.888 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-03-27T02:07:16.900+0000 7f3afaa33a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.888 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-03-27T02:07:16.900+0000 7f3afaa33a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.889 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:16.902+0000 7f4005634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.920 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:16.927+0000 7f9bfd71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:16.943 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:16.956+0000 7f9bfd71d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:17.015 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2024-03-27T02:07:17.026 INFO:tasks.ceph:check_enable_crimson: True 2024-03-27T02:07:17.026 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-27T02:07:17.026 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-27T02:07:17.093 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:17.105+0000 7f4005634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:17.093 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:17.105+0000 7f4005634000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:17.136 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:17.145+0000 7f6c8aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:17.139 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:17.151+0000 7f6c8aa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr:2024-03-27T02:07:17.236+0000 7f3af13f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr:Please be aware that this feature is experimental, untested, 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr:unsupported, and may result in data corruption, data loss, 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr:feature with important data. 2024-03-27T02:07:17.224 INFO:tasks.ceph.mon.a.smithi033.stderr: 2024-03-27T02:07:17.583 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:17.595+0000 7f4005634000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-27T02:07:17.638 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:17.651+0000 7f4005634000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-27T02:07:18.027 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-27T02:07:18.027 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:18.027 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-27T02:07:18.054 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:18.054 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-27T02:07:18.121 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:18.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-27T02:07:18.186 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.198+0000 7f4005634000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-27T02:07:18.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 939068d0-1562-4f6f-9994-79a3f1255c8f 0 2024-03-27T02:07:18.242 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.255+0000 7f4005634000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-27T02:07:18.290 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.302+0000 7f4005634000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-27T02:07:18.340 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.347+0000 7f7c90a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.342 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.355+0000 7f7c90a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.355 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.368+0000 7f4005634000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-27T02:07:18.400 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.412+0000 7f4005634000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-27T02:07:18.423 INFO:teuthology.orchestra.run.smithi033.stdout:0 2024-03-27T02:07:18.434 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 4e325b38-bbf3-404c-ad7e-49b7ac7035be 1 2024-03-27T02:07:18.523 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.535+0000 7f4005634000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-27T02:07:18.586 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.592+0000 7f968dcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.588 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.600+0000 7f968dcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.600 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.613+0000 7f4005634000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-27T02:07:18.648 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.661+0000 7f4005634000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-27T02:07:18.669 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-27T02:07:18.681 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 95df496a-a11a-4711-a7a9-c222e780dcc9 2 2024-03-27T02:07:18.704 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.717+0000 7f4005634000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-27T02:07:18.755 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:18.768+0000 7f4005634000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-27T02:07:18.793 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.800+0000 7fc96326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.795 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:18.807+0000 7fc96326d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:18.876 INFO:teuthology.orchestra.run.smithi033.stdout:2 2024-03-27T02:07:18.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-27T02:07:18.959 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-27T02:07:18.960 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-27T02:07:19.002 INFO:tasks.ceph.osd.0:Started 2024-03-27T02:07:19.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-27T02:07:19.036 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-27T02:07:19.036 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-27T02:07:19.079 INFO:tasks.ceph.osd.1:Started 2024-03-27T02:07:19.079 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-27T02:07:19.112 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-27T02:07:19.112 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-27T02:07:19.155 INFO:tasks.ceph.osd.2:Started 2024-03-27T02:07:19.155 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T02:07:19.176 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.189+0000 7f4005634000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-27T02:07:19.323 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:19.331+0000 7fbf8be13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:19.326 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:19.339+0000 7fbf8be13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:19.334 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.347+0000 7f4005634000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-27T02:07:19.398 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:19.398 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":6,"fsid":"80632d94-d178-4e29-aef4-920a112be4f8","created":"2024-03-27T02:07:16.949332+0000","modified":"2024-03-27T02:07:18.888486+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T02:07:18.888486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"939068d0-1562-4f6f-9994-79a3f1255c8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e325b38-bbf3-404c-ad7e-49b7ac7035be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"95df496a-a11a-4711-a7a9-c222e780dcc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T02:07:19.409 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-27T02:07:19.410 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-27T02:07:19.506 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.519+0000 7f4005634000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-27T02:07:19.555 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.567+0000 7f4005634000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-27T02:07:19.676 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.688+0000 7f4005634000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-27T02:07:19.712 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T02:07:19.781 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.794+0000 7f4005634000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-27T02:07:19.815 INFO:teuthology.misc.health.smithi033.stderr:2024-03-27T02:07:19.824+0000 7f0a19c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:19.817 INFO:teuthology.misc.health.smithi033.stderr:2024-03-27T02:07:19.830+0000 7f0a19c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:19.887 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-27T02:07:19.887 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":6,"fsid":"80632d94-d178-4e29-aef4-920a112be4f8","created":"2024-03-27T02:07:16.949332+0000","modified":"2024-03-27T02:07:18.888486+0000","last_up_change":"0.000000","last_in_change":"2024-03-27T02:07:18.888486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"939068d0-1562-4f6f-9994-79a3f1255c8f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4e325b38-bbf3-404c-ad7e-49b7ac7035be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"95df496a-a11a-4711-a7a9-c222e780dcc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T02:07:19.898 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-27T02:07:19.953 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:19.966+0000 7f4005634000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-27T02:07:20.001 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:20.013+0000 7f4005634000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-27T02:07:20.175 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:20.187+0000 7f4005634000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-27T02:07:21.042 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:21.054+0000 7f4005634000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-27T02:07:21.331 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:21.343+0000 7f4005634000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-27T02:07:21.385 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:21.397+0000 7f4005634000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-27T02:07:21.516 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:21.528+0000 7f4005634000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-27T02:07:22.537 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-27T02:07:22.550+0000 7f3fa4476640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-27T02:07:26.201 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-27T02:07:26.316 INFO:teuthology.misc.health.smithi033.stderr:2024-03-27T02:07:26.325+0000 7f156f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:26.318 INFO:teuthology.misc.health.smithi033.stderr:2024-03-27T02:07:26.330+0000 7f156f99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:26.512 INFO:teuthology.misc.health.smithi033.stdout: 2024-03-27T02:07:26.513 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":8,"fsid":"80632d94-d178-4e29-aef4-920a112be4f8","created":"2024-03-27T02:07:16.949332+0000","modified":"2024-03-27T02:07:25.549741+0000","last_up_change":"2024-03-27T02:07:25.549741+0000","last_in_change":"2024-03-27T02:07:18.888486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"939068d0-1562-4f6f-9994-79a3f1255c8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":1344507241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1344507241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":1344507241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6815","nonce":1344507241}]},"public_addr":"172.21.15.33:6807/1344507241","cluster_addr":"172.21.15.33:6806/1344507241","heartbeat_back_addr":"172.21.15.33:6814/1344507241","heartbeat_front_addr":"172.21.15.33:6815/1344507241","state":["exists","up"]},{"osd":1,"uuid":"4e325b38-bbf3-404c-ad7e-49b7ac7035be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2084331126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2084331126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2084331126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2084331126}]},"public_addr":"172.21.15.33:6804/2084331126","cluster_addr":"172.21.15.33:6805/2084331126","heartbeat_back_addr":"172.21.15.33:6811/2084331126","heartbeat_front_addr":"172.21.15.33:6808/2084331126","state":["exists","up"]},{"osd":2,"uuid":"95df496a-a11a-4711-a7a9-c222e780dcc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":2000727961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2000727961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":2000727961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2000727961}]},"public_addr":"172.21.15.33:6809/2000727961","cluster_addr":"172.21.15.33:6810/2000727961","heartbeat_back_addr":"172.21.15.33:6813/2000727961","heartbeat_front_addr":"172.21.15.33:6802/2000727961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T02:07:26.524 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-27T02:07:26.524 INFO:tasks.ceph:Creating RBD pool 2024-03-27T02:07:26.524 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-27T02:07:26.643 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:26.649+0000 7f02adc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:26.645 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:26.658+0000 7f02adc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:27.552 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2024-03-27T02:07:27.578 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-27T02:07:27.722 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:27.724+0000 7fd08eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:27.724 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:27.737+0000 7fd08eb5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:28.556 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-27T02:07:28.579 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-27T02:07:28.580 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-27T02:07:28.580 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-27T02:07:28.580 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-27T02:07:28.580 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-27T02:07:28.733 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:28.742+0000 7f9ef786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:28.735 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:28.747+0000 7f9ef786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:28.981 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:28.991 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":4026442692},{"type":"v1","addr":"172.21.15.33:6801","nonce":4026442692}]},"active_addr":"172.21.15.33:6801/4026442692","active_change":"2024-03-27T02:07:21.542128+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1901236447}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2711923112}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":3328479951}]}]} 2024-03-27T02:07:28.996 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-27T02:07:28.996 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-27T02:07:28.996 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T02:07:29.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.118+0000 7f94c135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.111 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.124+0000 7f94c135d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.307 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:29.308 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"fsid":"80632d94-d178-4e29-aef4-920a112be4f8","created":"2024-03-27T02:07:16.949332+0000","modified":"2024-03-27T02:07:28.569257+0000","last_up_change":"2024-03-27T02:07:25.549741+0000","last_in_change":"2024-03-27T02:07:18.888486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T02:07:25.573322+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T02:07:26.853626+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"939068d0-1562-4f6f-9994-79a3f1255c8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":1344507241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1344507241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":1344507241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6815","nonce":1344507241}]},"public_addr":"172.21.15.33:6807/1344507241","cluster_addr":"172.21.15.33:6806/1344507241","heartbeat_back_addr":"172.21.15.33:6814/1344507241","heartbeat_front_addr":"172.21.15.33:6815/1344507241","state":["exists","up"]},{"osd":1,"uuid":"4e325b38-bbf3-404c-ad7e-49b7ac7035be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2084331126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2084331126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2084331126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2084331126}]},"public_addr":"172.21.15.33:6804/2084331126","cluster_addr":"172.21.15.33:6805/2084331126","heartbeat_back_addr":"172.21.15.33:6811/2084331126","heartbeat_front_addr":"172.21.15.33:6808/2084331126","state":["exists","up"]},{"osd":2,"uuid":"95df496a-a11a-4711-a7a9-c222e780dcc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":2000727961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2000727961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":2000727961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2000727961}]},"public_addr":"172.21.15.33:6809/2000727961","cluster_addr":"172.21.15.33:6810/2000727961","heartbeat_back_addr":"172.21.15.33:6813/2000727961","heartbeat_front_addr":"172.21.15.33:6802/2000727961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T02:07:29.321 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-27T02:07:29.321 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-27T02:07:29.434 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.443+0000 7f4b1702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.436 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.448+0000 7f4b1702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.632 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:29.632 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":12,"fsid":"80632d94-d178-4e29-aef4-920a112be4f8","created":"2024-03-27T02:07:16.949332+0000","modified":"2024-03-27T02:07:29.573218+0000","last_up_change":"2024-03-27T02:07:25.549741+0000","last_in_change":"2024-03-27T02:07:18.888486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-27T02:07:25.573322+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-27T02:07:26.853626+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"939068d0-1562-4f6f-9994-79a3f1255c8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":1344507241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":1344507241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":1344507241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6815","nonce":1344507241}]},"public_addr":"172.21.15.33:6807/1344507241","cluster_addr":"172.21.15.33:6806/1344507241","heartbeat_back_addr":"172.21.15.33:6814/1344507241","heartbeat_front_addr":"172.21.15.33:6815/1344507241","state":["exists","up"]},{"osd":1,"uuid":"4e325b38-bbf3-404c-ad7e-49b7ac7035be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2084331126}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2084331126}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2084331126}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":2084331126}]},"public_addr":"172.21.15.33:6804/2084331126","cluster_addr":"172.21.15.33:6805/2084331126","heartbeat_back_addr":"172.21.15.33:6811/2084331126","heartbeat_front_addr":"172.21.15.33:6808/2084331126","state":["exists","up"]},{"osd":2,"uuid":"95df496a-a11a-4711-a7a9-c222e780dcc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":2000727961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2000727961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":2000727961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":2000727961}]},"public_addr":"172.21.15.33:6809/2000727961","cluster_addr":"172.21.15.33:6810/2000727961","heartbeat_back_addr":"172.21.15.33:6813/2000727961","heartbeat_front_addr":"172.21.15.33:6802/2000727961","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-27T02:07:29.645 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-27T02:07:29.646 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-27T02:07:29.646 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-27T02:07:29.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.773+0000 7f8cf36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.773+0000 7f7b5b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.766 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.773+0000 7f03eabe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.767 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.780+0000 7f7b5b66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.780+0000 7f8cf36ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.768 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.780+0000 7f03eabe0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.794 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:29.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-27T02:07:29.803 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:29.803 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-27T02:07:29.804 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:29.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-27T02:07:29.913 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.920+0000 7fe4bf95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.915 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.928+0000 7fe4bf95d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.971+0000 7fe0f8314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.973+0000 7f760015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.978+0000 7fe0f8314640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:29.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:29.979+0000 7f760015d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.117 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:30.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2024-03-27T02:07:30.130 DEBUG:teuthology.parallel:result is None 2024-03-27T02:07:30.162 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:30.167 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2024-03-27T02:07:30.175 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2024-03-27T02:07:30.175 DEBUG:teuthology.parallel:result is None 2024-03-27T02:07:30.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2024-03-27T02:07:30.180 DEBUG:teuthology.parallel:result is None 2024-03-27T02:07:30.180 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-27T02:07:30.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T02:07:30.334 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.343+0000 7fa2f4824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.336 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.348+0000 7fa2f4824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.528 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:30.528 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-27T02:07:30.540 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-27T02:07:29.802428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80884,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034636,"statfs":{"total":289910292480,"available":289827467264,"internally_reserved":0,"allocated":405504,"data_stored":1245350,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.683647+0000","last_change":"2024-03-27T02:07:28.683814+0000","last_active":"2024-03-27T02:07:28.683647+0000","last_peered":"2024-03-27T02:07:28.683647+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.681984+0000","last_became_peered":"2024-03-27T02:07:28.681984+0000","last_unstale":"2024-03-27T02:07:28.683647+0000","last_undegraded":"2024-03-27T02:07:28.683647+0000","last_fullsized":"2024-03-27T02:07:28.683647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.688964+0000","last_change":"2024-03-27T02:07:28.689242+0000","last_active":"2024-03-27T02:07:28.688964+0000","last_peered":"2024-03-27T02:07:28.688964+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.686930+0000","last_became_peered":"2024-03-27T02:07:28.686930+0000","last_unstale":"2024-03-27T02:07:28.688964+0000","last_undegraded":"2024-03-27T02:07:28.688964+0000","last_fullsized":"2024-03-27T02:07:28.688964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.682004+0000","last_change":"2024-03-27T02:07:28.682463+0000","last_active":"2024-03-27T02:07:28.682004+0000","last_peered":"2024-03-27T02:07:28.682004+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.677603+0000","last_became_peered":"2024-03-27T02:07:28.677603+0000","last_unstale":"2024-03-27T02:07:28.682004+0000","last_undegraded":"2024-03-27T02:07:28.682004+0000","last_fullsized":"2024-03-27T02:07:28.682004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.700261+0000","last_change":"2024-03-27T02:07:28.700436+0000","last_active":"2024-03-27T02:07:28.700261+0000","last_peered":"2024-03-27T02:07:28.700261+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.698257+0000","last_became_peered":"2024-03-27T02:07:28.698257+0000","last_unstale":"2024-03-27T02:07:28.700261+0000","last_undegraded":"2024-03-27T02:07:28.700261+0000","last_fullsized":"2024-03-27T02:07:28.700261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.657317+0000","last_change":"2024-03-27T02:07:28.657506+0000","last_active":"2024-03-27T02:07:28.657317+0000","last_peered":"2024-03-27T02:07:28.657317+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.653458+0000","last_became_peered":"2024-03-27T02:07:28.653458+0000","last_unstale":"2024-03-27T02:07:28.657317+0000","last_undegraded":"2024-03-27T02:07:28.657317+0000","last_fullsized":"2024-03-27T02:07:28.657317+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T02:07:29.583564+0000","last_change":"2024-03-27T02:07:28.629613+0000","last_active":"2024-03-27T02:07:29.583564+0000","last_peered":"2024-03-27T02:07:29.583564+0000","last_clean":"2024-03-27T02:07:29.583564+0000","last_became_active":"2024-03-27T02:07:28.625543+0000","last_became_peered":"2024-03-27T02:07:28.625543+0000","last_unstale":"2024-03-27T02:07:29.583564+0000","last_undegraded":"2024-03-27T02:07:29.583564+0000","last_fullsized":"2024-03-27T02:07:29.583564+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T02:07:29.582537+0000","last_change":"2024-03-27T02:07:28.636386+0000","last_active":"2024-03-27T02:07:29.582537+0000","last_peered":"2024-03-27T02:07:29.582537+0000","last_clean":"2024-03-27T02:07:29.582537+0000","last_became_active":"2024-03-27T02:07:28.629753+0000","last_became_peered":"2024-03-27T02:07:28.629753+0000","last_unstale":"2024-03-27T02:07:29.582537+0000","last_undegraded":"2024-03-27T02:07:29.582537+0000","last_fullsized":"2024-03-27T02:07:29.582537+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.623201+0000","last_change":"2024-03-27T02:07:28.623497+0000","last_active":"2024-03-27T02:07:28.623201+0000","last_peered":"2024-03-27T02:07:28.623201+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.621141+0000","last_became_peered":"2024-03-27T02:07:28.621141+0000","last_unstale":"2024-03-27T02:07:28.623201+0000","last_undegraded":"2024-03-27T02:07:28.623201+0000","last_fullsized":"2024-03-27T02:07:28.623201+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.582744+0000","last_change":"2024-03-27T02:07:27.639584+0000","last_active":"2024-03-27T02:07:28.582744+0000","last_peered":"2024-03-27T02:07:28.582744+0000","last_clean":"2024-03-27T02:07:28.582744+0000","last_became_active":"2024-03-27T02:07:27.633976+0000","last_became_peered":"2024-03-27T02:07:27.633976+0000","last_unstale":"2024-03-27T02:07:28.582744+0000","last_undegraded":"2024-03-27T02:07:28.582744+0000","last_fullsized":"2024-03-27T02:07:28.582744+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:26.557295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:26.557295+0000","last_clean_scrub_stamp":"2024-03-27T02:07:26.557295+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":21538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":611906,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":611906,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T02:07:30.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-27T02:07:30.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.669+0000 7fdd944ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.674+0000 7fdd944ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.855 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:30.856 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2024-03-27T02:07:30.867 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-27T02:07:29.802428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80884,"kb_used_data":396,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034636,"statfs":{"total":289910292480,"available":289827467264,"internally_reserved":0,"allocated":405504,"data_stored":1245350,"data_compressed":59912,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.683647+0000","last_change":"2024-03-27T02:07:28.683814+0000","last_active":"2024-03-27T02:07:28.683647+0000","last_peered":"2024-03-27T02:07:28.683647+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.681984+0000","last_became_peered":"2024-03-27T02:07:28.681984+0000","last_unstale":"2024-03-27T02:07:28.683647+0000","last_undegraded":"2024-03-27T02:07:28.683647+0000","last_fullsized":"2024-03-27T02:07:28.683647+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.688964+0000","last_change":"2024-03-27T02:07:28.689242+0000","last_active":"2024-03-27T02:07:28.688964+0000","last_peered":"2024-03-27T02:07:28.688964+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.686930+0000","last_became_peered":"2024-03-27T02:07:28.686930+0000","last_unstale":"2024-03-27T02:07:28.688964+0000","last_undegraded":"2024-03-27T02:07:28.688964+0000","last_fullsized":"2024-03-27T02:07:28.688964+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.682004+0000","last_change":"2024-03-27T02:07:28.682463+0000","last_active":"2024-03-27T02:07:28.682004+0000","last_peered":"2024-03-27T02:07:28.682004+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.677603+0000","last_became_peered":"2024-03-27T02:07:28.677603+0000","last_unstale":"2024-03-27T02:07:28.682004+0000","last_undegraded":"2024-03-27T02:07:28.682004+0000","last_fullsized":"2024-03-27T02:07:28.682004+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.700261+0000","last_change":"2024-03-27T02:07:28.700436+0000","last_active":"2024-03-27T02:07:28.700261+0000","last_peered":"2024-03-27T02:07:28.700261+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.698257+0000","last_became_peered":"2024-03-27T02:07:28.698257+0000","last_unstale":"2024-03-27T02:07:28.700261+0000","last_undegraded":"2024-03-27T02:07:28.700261+0000","last_fullsized":"2024-03-27T02:07:28.700261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.657317+0000","last_change":"2024-03-27T02:07:28.657506+0000","last_active":"2024-03-27T02:07:28.657317+0000","last_peered":"2024-03-27T02:07:28.657317+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.653458+0000","last_became_peered":"2024-03-27T02:07:28.653458+0000","last_unstale":"2024-03-27T02:07:28.657317+0000","last_undegraded":"2024-03-27T02:07:28.657317+0000","last_fullsized":"2024-03-27T02:07:28.657317+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T02:07:29.583564+0000","last_change":"2024-03-27T02:07:28.629613+0000","last_active":"2024-03-27T02:07:29.583564+0000","last_peered":"2024-03-27T02:07:29.583564+0000","last_clean":"2024-03-27T02:07:29.583564+0000","last_became_active":"2024-03-27T02:07:28.625543+0000","last_became_peered":"2024-03-27T02:07:28.625543+0000","last_unstale":"2024-03-27T02:07:29.583564+0000","last_undegraded":"2024-03-27T02:07:29.583564+0000","last_fullsized":"2024-03-27T02:07:29.583564+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-27T02:07:29.582537+0000","last_change":"2024-03-27T02:07:28.636386+0000","last_active":"2024-03-27T02:07:29.582537+0000","last_peered":"2024-03-27T02:07:29.582537+0000","last_clean":"2024-03-27T02:07:29.582537+0000","last_became_active":"2024-03-27T02:07:28.629753+0000","last_became_peered":"2024-03-27T02:07:28.629753+0000","last_unstale":"2024-03-27T02:07:29.582537+0000","last_undegraded":"2024-03-27T02:07:29.582537+0000","last_fullsized":"2024-03-27T02:07:29.582537+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.623201+0000","last_change":"2024-03-27T02:07:28.623497+0000","last_active":"2024-03-27T02:07:28.623201+0000","last_peered":"2024-03-27T02:07:28.623201+0000","last_clean":"2024-03-27T02:07:27.564966+0000","last_became_active":"2024-03-27T02:07:28.621141+0000","last_became_peered":"2024-03-27T02:07:28.621141+0000","last_unstale":"2024-03-27T02:07:28.623201+0000","last_undegraded":"2024-03-27T02:07:28.623201+0000","last_fullsized":"2024-03-27T02:07:28.623201+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:27.564966+0000","last_clean_scrub_stamp":"2024-03-27T02:07:27.564966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-27T02:07:28.582744+0000","last_change":"2024-03-27T02:07:27.639584+0000","last_active":"2024-03-27T02:07:28.582744+0000","last_peered":"2024-03-27T02:07:28.582744+0000","last_clean":"2024-03-27T02:07:28.582744+0000","last_became_active":"2024-03-27T02:07:27.633976+0000","last_became_peered":"2024-03-27T02:07:27.633976+0000","last_unstale":"2024-03-27T02:07:28.582744+0000","last_undegraded":"2024-03-27T02:07:28.582744+0000","last_fullsized":"2024-03-27T02:07:28.582744+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-27T02:07:26.557295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-27T02:07:26.557295+0000","last_clean_scrub_stamp":"2024-03-27T02:07:26.557295+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":21538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":611906,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26984,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344856,"statfs":{"total":96636764160,"available":96609132544,"internally_reserved":0,"allocated":155648,"data_stored":611906,"data_compressed":29956,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-27T02:07:30.868 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-27T02:07:30.868 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-27T02:07:30.868 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-27T02:07:30.868 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-27T02:07:30.981 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.991+0000 7f974375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:30.983 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:07:30.995+0000 7f974375d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:07:31.217 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:31.217 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-27T02:07:31.230 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-27T02:07:31.230 INFO:teuthology.run_tasks:Running task cbt... 2024-03-27T02:07:31.243 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi033.front.sepia.ceph.com', 'osds': ['ubuntu@smithi033.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi033.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi033.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi033.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-03-27T02:07:31.243 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-03-27T02:07:31.260 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:07:31.260 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-27T02:07:31.314 DEBUG:teuthology.orchestra.run.smithi033:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-03-27T02:07:31.369 INFO:teuthology.orchestra.run.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-03-27T02:07:31.711 INFO:tasks.cbt:Installing collectl 2024-03-27T02:07:31.712 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-03-27T02:07:31.817 INFO:teuthology.orchestra.run.smithi033.stderr:--2024-03-27 02:07:31-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-03-27T02:07:31.963 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2024-03-27T02:07:31.977 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-03-27T02:07:32.483 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-27T02:07:32.483 INFO:teuthology.orchestra.run.smithi033.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmA39kxSY1UTadHLi3urNhBXOzkQMjeIvW6q8IscsxlKfh0OLBH22klD4CxAkTMd5TotRPyJlzzk5b828CEYctTBXTjw%3D%3D&use_mirror=versaweb&r= [following] 2024-03-27T02:07:32.483 INFO:teuthology.orchestra.run.smithi033.stderr:--2024-03-27 02:07:32-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmA39kxSY1UTadHLi3urNhBXOzkQMjeIvW6q8IscsxlKfh0OLBH22klD4CxAkTMd5TotRPyJlzzk5b828CEYctTBXTjw%3D%3D&use_mirror=versaweb&r= 2024-03-27T02:07:32.483 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-03-27T02:07:32.548 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-03-27T02:07:32.802 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-27T02:07:32.802 INFO:teuthology.orchestra.run.smithi033.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-03-27T02:07:32.802 INFO:teuthology.orchestra.run.smithi033.stderr:--2024-03-27 02:07:32-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-03-27T02:07:32.814 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2024-03-27T02:07:32.891 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2024-03-27T02:07:33.308 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-27T02:07:33.308 INFO:teuthology.orchestra.run.smithi033.stderr:Length: 620693 (606K) [application/x-gzip] 2024-03-27T02:07:33.308 INFO:teuthology.orchestra.run.smithi033.stderr:Saving to: ‘collectl.tar.gz’ 2024-03-27T02:07:33.308 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:33.410 INFO:teuthology.orchestra.run.smithi033.stderr: 0K .......... .......... .......... .......... .......... 8% 488K 1s 2024-03-27T02:07:33.485 INFO:teuthology.orchestra.run.smithi033.stderr: 50K .......... .......... .......... .......... .......... 16% 682K 1s 2024-03-27T02:07:33.551 INFO:teuthology.orchestra.run.smithi033.stderr: 100K .......... .......... .......... .......... .......... 24% 737K 1s 2024-03-27T02:07:33.610 INFO:teuthology.orchestra.run.smithi033.stderr: 150K .......... .......... .......... .......... .......... 32% 849K 1s 2024-03-27T02:07:33.680 INFO:teuthology.orchestra.run.smithi033.stderr: 200K .......... .......... .......... .......... .......... 41% 717K 1s 2024-03-27T02:07:33.747 INFO:teuthology.orchestra.run.smithi033.stderr: 250K .......... .......... .......... .......... .......... 49% 741K 0s 2024-03-27T02:07:33.820 INFO:teuthology.orchestra.run.smithi033.stderr: 300K .......... .......... .......... .......... .......... 57% 691K 0s 2024-03-27T02:07:33.905 INFO:teuthology.orchestra.run.smithi033.stderr: 350K .......... .......... .......... .......... .......... 65% 585K 0s 2024-03-27T02:07:33.973 INFO:teuthology.orchestra.run.smithi033.stderr: 400K .......... .......... .......... .......... .......... 74% 737K 0s 2024-03-27T02:07:34.041 INFO:teuthology.orchestra.run.smithi033.stderr: 450K .......... .......... .......... .......... .......... 82% 736K 0s 2024-03-27T02:07:34.100 INFO:teuthology.orchestra.run.smithi033.stderr: 500K .......... .......... .......... .......... .......... 90% 848K 0s 2024-03-27T02:07:34.169 INFO:teuthology.orchestra.run.smithi033.stderr: 550K .......... .......... .......... .......... .......... 98% 730K 0s 2024-03-27T02:07:34.169 INFO:teuthology.orchestra.run.smithi033.stderr: 600K ...... 100% 11.4T=0.9s 2024-03-27T02:07:34.169 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:34.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27 02:07:34 (704 KB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-03-27T02:07:34.169 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:07:34.199 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/ 2024-03-27T02:07:34.199 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/collectl.conf 2024-03-27T02:07:34.199 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ 2024-03-27T02:07:34.199 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/SlowProc.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/WhySummary.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Socket.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/colmux.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Tutorial.html 2024-03-27T02:07:34.200 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Gexpr.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Import.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/style.css 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/FileNaming.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/TheMath.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Environmental.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Process.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/HiResTime.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/slub.jpg 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Lustre.html 2024-03-27T02:07:34.209 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Documentation.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Inodes.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Graphite.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/InputFiles.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Export.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Interrupts.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Startup.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Performance.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Messages.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Logging.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-detail.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/CPUs.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Matrix.html 2024-03-27T02:07:34.210 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-brief.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Misc.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Examples.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OpenStack.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Exceptions.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Playback.html 2024-03-27T02:07:34.211 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Hello.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Sockets.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Network.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Infiniband.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Memory.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Disks.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/service/ 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/service/collectl.service 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/ 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-suse 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-debian 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-generic 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl 2024-03-27T02:07:34.212 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/misc.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/gexpr.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/statsd.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/envrules.std 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/hello.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vmsum.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/proctree.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/README-WINDOWS 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/formatit.ph 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/colmux 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/INSTALL 2024-03-27T02:07:34.213 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/graphite.ph 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/RELEASE-collectl 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vmstat.ph 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/README 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/ 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/collectl.1 2024-03-27T02:07:34.214 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/colmux.1 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/lexpr.ph 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/client.pl 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/collectl 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/COPYING 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vnet.ph 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/GPL 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/UNINSTALL 2024-03-27T02:07:34.215 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/ARTISTIC 2024-03-27T02:07:34.267 INFO:teuthology.orchestra.run.smithi033.stderr:./INSTALL: line 90: chkconfig: command not found 2024-03-27T02:07:34.267 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-03-27T02:07:34.854 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:14 ago on Wed 27 Mar 2024 02:04:20 AM UTC. 2024-03-27T02:07:34.957 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-03-27T02:07:34.959 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-03-27T02:07:35.018 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout: librbd-devel x86_64 2:19.0.0-2463.g4254c927.el9 ceph 23 k 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-03-27T02:07:35.019 INFO:teuthology.orchestra.run.smithi033.stdout: perf x86_64 5.14.0-432.el9 appstream 2.0 M 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout: libradospp-devel x86_64 2:19.0.0-2463.g4254c927.el9 ceph 30 k 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout:Install 5 Packages 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:35.020 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.2 M 2024-03-27T02:07:35.021 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 11 M 2024-03-27T02:07:35.021 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-27T02:07:35.497 INFO:teuthology.orchestra.run.smithi033.stdout:(1/5): librbd-devel-19.0.0-2463.g4254c927.el9.x 140 kB/s | 23 kB 00:00 2024-03-27T02:07:35.589 INFO:teuthology.orchestra.run.smithi033.stdout:(2/5): libradospp-devel-19.0.0-2463.g4254c927.e 118 kB/s | 30 kB 00:00 2024-03-27T02:07:35.772 INFO:teuthology.orchestra.run.smithi033.stdout:(3/5): perf-5.14.0-432.el9.x86_64.rpm 4.6 MB/s | 2.0 MB 00:00 2024-03-27T02:07:35.822 INFO:teuthology.orchestra.run.smithi033.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 56 kB/s | 13 kB 00:00 2024-03-27T02:07:35.856 INFO:teuthology.orchestra.run.smithi033.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 330 kB/s | 118 kB 00:00 2024-03-27T02:07:35.858 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-27T02:07:35.859 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-03-27T02:07:35.886 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-27T02:07:35.909 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-27T02:07:35.909 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-27T02:07:36.030 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-27T02:07:36.031 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-27T02:07:36.324 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-27T02:07:36.404 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-03-27T02:07:36.439 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-03-27T02:07:36.470 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradospp-devel-2:19.0.0-2463.g4254c927.el9.x86_64 3/5 2024-03-27T02:07:36.811 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librbd-devel-2:19.0.0-2463.g4254c927.el9.x86_64 4/5 2024-03-27T02:07:36.887 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : perf-5.14.0-432.el9.x86_64 5/5 2024-03-27T02:07:37.917 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: perf-5.14.0-432.el9.x86_64 5/5 2024-03-27T02:07:37.918 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradospp-devel-2:19.0.0-2463.g4254c927.el9.x86_64 1/5 2024-03-27T02:07:37.918 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd-devel-2:19.0.0-2463.g4254c927.el9.x86_64 2/5 2024-03-27T02:07:37.918 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : perf-5.14.0-432.el9.x86_64 3/5 2024-03-27T02:07:37.918 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-03-27T02:07:38.297 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-03-27T02:07:38.297 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: libradospp-devel-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: librbd-devel-2:19.0.0-2463.g4254c927.el9.x86_64 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-2.34-7.el9.x86_64 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: perf-5.14.0-432.el9.x86_64 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-27T02:07:38.298 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-27T02:07:38.690 INFO:tasks.cbt:benchmark: radosbench 2024-03-27T02:07:38.690 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-03-27T02:07:38.718 DEBUG:teuthology.orchestra.run.smithi033:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - Settings.cluster: 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: 'clients': ['ubuntu@smithi033.front.sepia.ceph.com'], 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: 'head': 'ubuntu@smithi033.front.sepia.ceph.com', 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: 'is_teuthology': True, 2024-03-27T02:07:39.103 INFO:teuthology.orchestra.run.smithi033.stderr: 'iterations': 1, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'mons': ['ubuntu@smithi033.front.sepia.ceph.com'], 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'osds': ['ubuntu@smithi033.front.sepia.ceph.com'], 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'osds_per_node': 3, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'pid_dir': '/var/run/ceph', 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'pgp_size': 256, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'replication': 'replicated'}}, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'rebuild_every_test': False, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'rgws': {'ubuntu@smithi033.front.sepia.ceph.com': None}, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'tmp_dir': '/tmp/cbt', 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'use_existing': True, 2024-03-27T02:07:39.104 INFO:teuthology.orchestra.run.smithi033.stderr: 'user': 'ubuntu'} 2024-03-27T02:07:39.125 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-03-27T02:07:39.126 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.126 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.126 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-03-27T02:07:39.129 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.129 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-27T02:07:39.131 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.131 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.131 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-03-27T02:07:39.133 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:07:39.133 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-27T02:07:39.133 INFO:teuthology.orchestra.run.smithi033.stderr:02:07:39 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-27T02:08:39.196 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.196 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-27T02:08:39.244 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.244 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-27T02:08:39.288 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.288 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-03-27T02:08:39.289 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.289 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-03-27T02:08:39.291 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.291 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.291 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-03-27T02:08:39.308 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.308 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/$(basename ${i}).smithi033.front.sepia.ceph.com ; done 2024-03-27T02:08:39.312 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.312 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.312 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-03-27T02:08:39.350 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - WARNING - cbt - sudo killall -9 rados 2024-03-27T02:08:39.351 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - WARNING - cbt - error 1 seen, continuing anyway... 2024-03-27T02:08:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d. 2024-03-27T02:08:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-03-27T02:08:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.370 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.371 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-03-27T02:08:39.386 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.386 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.387 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-03-27T02:08:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-27T02:08:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.391 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.391 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.394 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-03-27T02:08:39.713 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.713 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:39.713 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-03-27T02:08:40.643 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:40 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:40.643 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:40 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:40.643 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-03-27T02:08:41.657 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:41 - INFO - cbt - Checking Health after pool creation. 2024-03-27T02:08:46.663 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:46 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:46.663 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:46 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:46.663 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-27T02:08:47.157 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:47 - INFO - cbt - Final Pool Health Check. 2024-03-27T02:08:52.163 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.163 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.164 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-27T02:08:52.499 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.499 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-27T02:08:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.524 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-27T02:08:52.561 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.561 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-03-27T02:08:52.563 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.563 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-03-27T02:08:52.565 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.565 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.565 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-27T02:08:52.671 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:52.671 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-27T02:08:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.243 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-03-27T02:08:53.588 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.588 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.588 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-03-27T02:08:53.651 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.651 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:53.651 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-03-27T02:08:54.463 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.464 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.464 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-03-27T02:08:54.466 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.466 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-27T02:08:54.467 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.467 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.467 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - INFO - cbt - Running radosbench write test. 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.471 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-03-27T02:08:54.472 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.472 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:08:54.472 INFO:teuthology.orchestra.run.smithi033.stderr:02:08:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-03-27T02:09:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.635 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-27T02:09:55.665 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.665 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-27T02:09:55.703 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.704 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-03-27T02:09:55.705 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.705 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-03-27T02:09:55.706 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.707 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:55.707 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-03-27T02:09:56.018 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.022 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.023 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-03-27T02:09:56.038 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.038 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/$(basename ${i}).smithi033.front.sepia.ceph.com ; done 2024-03-27T02:09:56.075 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - INFO - cbt - Convert results to json format. 2024-03-27T02:09:56.075 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.076 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.076 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.076 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-27T02:09:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.357 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-27T02:09:56.640 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.641 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.641 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-03-27T02:09:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:56.985 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq 2024-03-27T02:09:57.023 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.024 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.024 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/ceph_settings.out 2024-03-27T02:09:57.850 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.850 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.851 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl 2024-03-27T02:09:57.853 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.853 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-27T02:09:57.854 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.854 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.854 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf 2024-03-27T02:09:57.857 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - INFO - cbt - Running radosbench seq test. 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.858 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.0 2024-03-27T02:09:57.859 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.859 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:09:57.859 INFO:teuthology.orchestra.run.smithi033.stderr:02:09:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/output.1 2024-03-27T02:10:14.987 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:14.987 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-27T02:10:15.021 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.022 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-27T02:10:15.061 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.061 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf.data 2024-03-27T02:10:15.063 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.063 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/perf/perf_stat.* 2024-03-27T02:10:15.064 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.064 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.064 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/historic_ops.out 2024-03-27T02:10:15.376 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.376 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.376 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* 2024-03-27T02:10:15.392 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.392 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/seq/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/$(basename ${i}).smithi033.front.sepia.ceph.com ; done 2024-03-27T02:10:15.408 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - INFO - cbt - Convert results to json format. 2024-03-27T02:10:15.408 INFO:teuthology.orchestra.run.smithi033.stderr:02:10:15 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.427 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-03-27T02:10:15.436 DEBUG:teuthology.orchestra.run.smithi033:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-03-27T02:10:15.559 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-03-27T02:10:15.559 INFO:tasks.cbt_performance:reading cbt preferences 2024-03-27T02:10:15.559 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-03-27T02:10:15.623 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.624 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.624 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.624 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi033.front.sepia.ceph.com 2024-03-27T02:10:15.625 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:10:15.625 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.1.smithi033.front.sepia.ceph.com of=/dev/stdout 2024-03-27T02:10:15.648 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-27T02:10:15.648 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2024-03-27T02:10:15.703 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-27T02:10:15.703 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-27T02:10:15.703 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:10:15.703 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write/json_output.0.smithi033.front.sepia.ceph.com of=/dev/stdout 2024-03-27T02:10:15.764 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-27T02:10:15.764 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/write -name 'perf_stat.*' 2024-03-27T02:10:15.819 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-27T02:10:15.819 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-27T02:10:15.819 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:10:15.819 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.1.smithi033.front.sepia.ceph.com of=/dev/stdout 2024-03-27T02:10:15.873 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-27T02:10:15.874 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2024-03-27T02:10:15.928 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-27T02:10:15.929 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-27T02:10:15.929 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-27T02:10:15.929 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq/json_output.0.smithi033.front.sepia.ceph.com of=/dev/stdout 2024-03-27T02:10:15.983 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-27T02:10:15.984 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-839a075d/seq -name 'perf_stat.*' 2024-03-27T02:10:16.038 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-27T02:10:16.038 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-27T02:10:16.049 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-27T02:10:16.056 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-27T02:10:16.062 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-27T02:10:16.068 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-27T02:10:16.069 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-27T02:10:16.078 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-27T02:10:16.209 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:10:16.214+0000 7fdff7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:10:16.211 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-27T02:10:16.223+0000 7fdff7a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-27T02:10:16.426 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-27T02:10:16.426 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-27T02:10:16.426 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-27T02:10:16.426 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T02:10:19.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-03-27T02:10:22.529 INFO:tasks.ceph.osd.0:Stopped 2024-03-27T02:10:22.529 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-27T02:10:22.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T02:10:24.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-03-27T02:10:24.891 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-03-27T02:10:28.630 INFO:tasks.ceph.osd.1:Stopped 2024-03-27T02:10:28.631 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-27T02:10:28.631 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T02:10:30.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-03-27T02:10:30.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-03-27T02:10:34.733 INFO:tasks.ceph.osd.2:Stopped 2024-03-27T02:10:34.733 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-27T02:10:34.734 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-27T02:10:34.734 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T02:10:34.766 INFO:tasks.ceph.mgr.x:Stopped 2024-03-27T02:10:34.766 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-27T02:10:34.767 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-27T02:10:34.767 INFO:teuthology.orchestra.run:waiting for 300 2024-03-27T02:10:34.820 INFO:tasks.ceph.mon.a:Stopped 2024-03-27T02:10:34.820 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-27T02:10:34.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-27T02:10:34.919 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:34.919 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-27T02:10:35.094 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-03-27T02:10:35.098 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:35.098 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-27T02:10:35.183 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:35.183 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-27T02:10:35.240 INFO:tasks.ceph:Archiving mon data... 2024-03-27T02:10:35.241 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/data/mon.a.tgz 2024-03-27T02:10:35.241 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-27T02:10:35.276 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.inl2INEuFH 2024-03-27T02:10:35.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.inl2INEuFH 2024-03-27T02:10:35.403 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.inl2INEuFH 2024-03-27T02:10:35.507 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.inl2INEuFH is 186KB 2024-03-27T02:10:35.559 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.inl2INEuFH 2024-03-27T02:10:35.574 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-27T02:10:35.574 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-27T02:10:35.707 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-27T02:10:35.707 INFO:tasks.ceph:Archiving crash dumps... 2024-03-27T02:10:35.708 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/remote/smithi033/crash 2024-03-27T02:10:35.709 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-27T02:10:35.772 INFO:tasks.ceph:Compressing logs... 2024-03-27T02:10:35.772 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30283.log 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31328.log 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32376.log 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30243.log 2024-03-27T02:10:35.847 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30934.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30005.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30976.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31457.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31037.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31457.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28646.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32804.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph.tmp-client.admin.28646.log: -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30878.log 2024-03-27T02:10:35.857 INFO:teuthology.orchestra.run.smithi033.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.32804.log.gz 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31499.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30878.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32376.log.gz 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33106.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31203.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.33106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32502.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33106.log.gz 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30203.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32478.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31201.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32589.log 2024-03-27T02:10:35.858 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32478.log: /var/log/ceph/ceph-client.admin.31201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32441.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30471.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32441.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.32589.log.gz 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32590.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32441.log.gz 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31121.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31415.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr: 66.4%/var/log/ceph/ceph-client.admin.31121.log: -- replaced with /var/log/ceph/ceph-client.admin.32590.log.gz 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31202.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31415.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30441.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30441.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32525.log 2024-03-27T02:10:35.859 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31079.log 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30093.log 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32805.log 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.30093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31289.log 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30093.log.gz 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.32805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31329.log 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.32805.log.gz 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.31289.log: /var/log/ceph/ceph-client.admin.31329.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30283.log.gz 2024-03-27T02:10:35.860 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30005.log.gz 2024-03-27T02:10:35.861 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30203.log.gz 2024-03-27T02:10:35.870 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30878.log.gz 2024-03-27T02:10:35.870 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30934.log.gz 2024-03-27T02:10:35.870 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30976.log.gz 2024-03-27T02:10:35.870 INFO:teuthology.orchestra.run.smithi033.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31037.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31079.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31121.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31201.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31203.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31202.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31289.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31328.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31329.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31415.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31457.log.gz 2024-03-27T02:10:35.871 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31499.log.gz 2024-03-27T02:10:35.887 INFO:teuthology.orchestra.run.smithi033.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.32478.log.gz 2024-03-27T02:10:35.889 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30471.log.gz 2024-03-27T02:10:35.910 INFO:teuthology.orchestra.run.smithi033.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28646.log.gz 2024-03-27T02:10:35.914 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32502.log.gz 2024-03-27T02:10:35.915 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32525.log.gz 2024-03-27T02:10:35.971 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30441.log.gz 2024-03-27T02:10:35.973 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30243.log.gz 2024-03-27T02:10:36.006 INFO:teuthology.orchestra.run.smithi033.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-27T02:10:36.012 INFO:teuthology.orchestra.run.smithi033.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-27T02:10:44.726 INFO:teuthology.orchestra.run.smithi033.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-27T02:10:45.419 INFO:teuthology.orchestra.run.smithi033.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-27T02:10:45.627 INFO:teuthology.orchestra.run.smithi033.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-27T02:10:45.629 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-27T02:10:45.629 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m9.802s 2024-03-27T02:10:45.629 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m5.874s 2024-03-27T02:10:45.630 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.300s 2024-03-27T02:10:45.630 INFO:tasks.ceph:Archiving logs... 2024-03-27T02:10:45.630 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/remote/smithi033/log 2024-03-27T02:10:45.632 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-27T02:10:46.573 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-27T02:10:46.584 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-27T02:10:46.585 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-27T02:10:46.643 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-27T02:10:46.653 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-27T02:10:46.653 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-27T02:10:46.675 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-27T02:10:46.743 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-27T02:10:46.744 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-27T02:10:46.744 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv01.front.sepia.ceph.com 2 6 377 21 +392us[ +396us] +/- 73ms 2024-03-27T02:10:46.744 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -252us[ -252us] +/- 57ms 2024-03-27T02:10:46.744 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -90us[ -90us] +/- 77ms 2024-03-27T02:10:46.744 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-27T02:10:46.749 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-27T02:10:46.762 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-27T02:10:46.762 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-27T02:10:46.791 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-27T02:10:46.929 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-27T02:10:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:10:46.959 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-27T02:10:46.969 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-27T02:10:46.993 INFO:teuthology.task.internal:Duration was 673.894349 seconds 2024-03-27T02:10:46.993 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-27T02:10:47.034 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-27T02:10:47.034 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-27T02:10:47.190 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-27T02:10:47.469 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-27T02:10:47.469 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-03-27T02:10:47.470 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-27T02:10:47.489 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-27T02:10:47.490 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-27T02:10:47.560 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-27T02:10:47.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-27T02:10:47.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-27T02:10:47.712 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-27T02:10:47.713 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-27T02:10:47.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-27T02:10:47.762 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-27T02:10:47.829 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-03-27T02:10:47.846 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-27T02:10:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-27T02:10:47.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-27T02:10:47.911 INFO:teuthology.task.internal:Transferring archived files... 2024-03-27T02:10:47.911 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-03-27_01:32:53-crimson-rados-ci-yingxin-crimson-load-obc-2-distro-default-smithi/7625043/remote/smithi033 2024-03-27T02:10:47.912 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-27T02:10:48.060 INFO:teuthology.task.internal:Removing archive directory... 2024-03-27T02:10:48.061 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-27T02:10:48.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-27T02:10:48.120 INFO:teuthology.task.internal:Not uploading archives. 2024-03-27T02:10:48.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-27T02:10:48.167 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-27T02:10:48.167 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-27T02:10:48.182 INFO:teuthology.orchestra.run.smithi033.stdout: 265312 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 27 02:10 /home/ubuntu/cephtest 2024-03-27T02:10:48.183 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-27T02:10:48.213 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-27T02:10:48.295 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-2&var-job_id_selected=7625043 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} duration: 673.894348859787 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-03-27T02:10:48.296 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-27T02:10:48.373 INFO:teuthology.run:pass