2024-09-22T03:29:03.145 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:29:03.169 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:29:03.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_write} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914359' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi060.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBClHYyBnZYLT34NhrwRaTFI7aZmDWB66VjtrXRLtFRaCIMtJtKxvIYxOlKSKyPbR4puycqxsA1yzZDF4XrVKVrA= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:29:03.256 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-22T03:29:03.256 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-22T03:29:03.257 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:29:03.258 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:29:03.279 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-22T03:29:03.279 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:29:03.279 INFO:teuthology.packaging:ref: None 2024-09-22T03:29:03.279 INFO:teuthology.packaging:tag: None 2024-09-22T03:29:03.279 INFO:teuthology.packaging:branch: main 2024-09-22T03:29:03.279 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:29:03.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-22T03:29:03.478 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-22T03:29:03.480 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:29:03.486 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:29:03.486 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:29:03.536 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:29:03.551 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:29:03.558 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:29:03.583 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi060.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:24:27.128756', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBClHYyBnZYLT34NhrwRaTFI7aZmDWB66VjtrXRLtFRaCIMtJtKxvIYxOlKSKyPbR4puycqxsA1yzZDF4XrVKVrA='} 2024-09-22T03:29:03.583 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:29:03.589 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:29:03.589 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:29:03.666 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8730fbc160>, signals=[15]) 2024-09-22T03:29:03.666 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:29:03.672 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:29:03.672 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:29:03.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:29:03.747 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:29:03.753 DEBUG:teuthology.orchestra.run.smithi060:> uname -m 2024-09-22T03:29:03.832 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2024-09-22T03:29:03.833 DEBUG:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2024-09-22T03:29:03.890 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="CentOS Stream" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="9" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:ID="centos" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="9" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:ANSI_COLOR="0;31" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:29:03.891 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:29:03.892 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:29:03.892 INFO:teuthology.orchestra.run.smithi060.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:29:03.892 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2024-09-22T03:29:03.911 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:29:03.919 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:29:03.972 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:29:03.972 DEBUG:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:29:03.989 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:29:04.010 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:29:04.011 DEBUG:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:29:04.055 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:29:04.106 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:29:04.132 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:29:04.132 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:29:04.133 DEBUG:teuthology.orchestra.run.smithi060:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:29:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:29:04.151 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-22T03:29:04.209 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:29:04.209 INFO:teuthology.task.kernel:Running kernel on smithi060: 5.14.0-503.el9.x86_64 2024-09-22T03:29:04.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum install -y kernel 2024-09-22T03:29:06.563 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-22T03:29:08.289 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 33 MB/s | 20 MB 00:00 2024-09-22T03:29:12.832 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-22T03:29:14.591 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 19 kB 00:00 2024-09-22T03:29:16.170 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-22T03:29:21.767 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-22T03:29:23.732 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:29:23.733 INFO:teuthology.orchestra.run.smithi060.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:29:23.827 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:29:23.832 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:29:23.833 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:29:23.834 INFO:teuthology.orchestra.run.smithi060.stdout:Install 4 Packages 2024-09-22T03:29:23.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:29:23.834 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 79 M 2024-09-22T03:29:23.834 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 126 M 2024-09-22T03:29:23.834 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:29:24.161 INFO:teuthology.orchestra.run.smithi060.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 118 kB/s | 36 kB 00:00 2024-09-22T03:29:32.135 INFO:teuthology.orchestra.run.smithi060.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.2 MB/s | 35 MB 00:08 2024-09-22T03:29:32.819 INFO:teuthology.orchestra.run.smithi060.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 3.3 MB/s | 29 MB 00:08 2024-09-22T03:29:32.977 INFO:teuthology.orchestra.run.smithi060.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 1.7 MB/s | 16 MB 00:09 2024-09-22T03:29:32.978 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:29:32.978 INFO:teuthology.orchestra.run.smithi060.stdout:Total 8.6 MB/s | 79 MB 00:09 2024-09-22T03:29:33.690 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:29:34.063 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:29:34.063 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:29:35.067 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:29:35.067 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:29:37.610 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:29:38.299 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:29:38.380 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:29:41.130 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:29:41.258 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:29:46.391 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:29:46.434 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:29:51.575 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:30:31.194 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:30:31.236 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:30:31.726 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:30:31.726 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:30:31.726 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:30:31.726 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:30:32.594 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:30:32.594 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:30:32.594 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:30:32.595 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:30:32.939 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:30:33.026 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:30:33.026 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:30:33.026 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:30:33.026 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:30:33.026 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:30:33.027 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:30:33.027 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi060.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:30:33.027 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:30:33.085 INFO:teuthology.orchestra.run.smithi060.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:30:33.085 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:30:33.085 DEBUG:teuthology.orchestra.run.smithi060:> sudo rpm -qi grub2-tools 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Name : grub2-tools 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Epoch : 1 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Version : 2.06 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Release : 82.el9 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: x86_64 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Group : Unspecified 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Size : 8274670 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:License : GPLv3+ 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:30:33.119 INFO:teuthology.orchestra.run.smithi060.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Packager : builder@centos.org 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Vendor : CentOS 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Summary : Support tools for GRUB. 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:Description : 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:30:33.120 INFO:teuthology.orchestra.run.smithi060.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:30:33.121 INFO:teuthology.orchestra.run.smithi060.stdout:hardware devices. 2024-09-22T03:30:33.121 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:30:33.121 INFO:teuthology.orchestra.run.smithi060.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:30:33.122 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:30:33.122 INFO:teuthology.task.kernel:Updating grub on smithi060 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:30:33.122 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:30:33.844 INFO:teuthology.orchestra.run.smithi060.stderr:Generating grub configuration file ... 2024-09-22T03:30:35.179 INFO:teuthology.orchestra.run.smithi060.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:30:35.193 INFO:teuthology.orchestra.run.smithi060.stderr:done 2024-09-22T03:30:35.195 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-22T03:30:35.211 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.60rPwh3Qid 2024-09-22T03:30:35.211 DEBUG:teuthology.orchestra.run.smithi060:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.60rPwh3Qid 2024-09-22T03:30:35.277 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.60rPwh3Qid 2024-09-22T03:30:35.409 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.60rPwh3Qid is 6KB 2024-09-22T03:30:35.457 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.60rPwh3Qid 2024-09-22T03:30:35.471 DEBUG:teuthology.orchestra.run.smithi060:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:30:35.533 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:30:35.533 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:30:35.533 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:30:35.533 INFO:teuthology.orchestra.run.smithi060.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:30:35.534 DEBUG:teuthology.orchestra.run.smithi060:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:30:35.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo shutdown -r now 2024-09-22T03:31:05.718 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:31:05.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-22T03:31:05.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:32:05.779 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:32:14.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-22T03:32:14.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:32:15.123 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-22T03:32:15.486 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-22T03:32:15.486 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:32:15.486 DEBUG:teuthology.orchestra.run.smithi060:> uname -r 2024-09-22T03:32:15.501 INFO:teuthology.orchestra.run.smithi060.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:32:15.502 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:32:15.502 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:32:15.502 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:32:16.502 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:32:16.503 DEBUG:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:32:16.611 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2024-09-22T03:32:16.639 DEBUG:teuthology.parallel:result is None 2024-09-22T03:32:16.639 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:32:16.649 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:32:16.649 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:32:16.664 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:32:16.703 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:32:16.733 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:32:16.734 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:32:16.754 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:32:16.761 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:32:16.762 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:32:16.820 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:32:16.832 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:32:16.833 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:32:16.841 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:32:16.842 DEBUG:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:32:16.897 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:32:16.913 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:32:16.913 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:32:16.949 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:32:17.031 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:32:17.086 DEBUG:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:32:17.165 DEBUG:teuthology.orchestra.run.smithi060:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:32:17.203 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:32:17.203 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:32:17.266 DEBUG:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2024-09-22T03:32:17.395 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:32:17.617 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:32:17.625 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:32:17.625 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:32:17.653 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:32:17.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo service auditd rotate 2024-09-22T03:32:17.780 INFO:teuthology.orchestra.run.smithi060.stdout:Rotating logs: 2024-09-22T03:32:17.782 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:32:17.782 DEBUG:teuthology.orchestra.run.smithi060:> /usr/sbin/getenforce 2024-09-22T03:32:17.820 INFO:teuthology.orchestra.run.smithi060.stdout:Permissive 2024-09-22T03:32:17.820 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi060.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:32:17.821 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T03:32:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:32:17.883 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:32:17.883 DEBUG:teuthology.orchestra.run.smithi060:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:32:17.951 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:32:17.971 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:32:17.982 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:32:17.983 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:36:04.172 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2024-09-22T03:36:04.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-22T03:36:04.174 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:36:04.247 DEBUG:teuthology.orchestra.run.smithi060:> true 2024-09-22T03:36:04.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi060.front.sepia.ceph.com' 2024-09-22T03:36:04.330 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:36:04.339 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:36:04.339 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:36:04.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:36:04.400 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:36:04.415 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:36:04.441 INFO:teuthology.orchestra.run.smithi060.stderr:sudo: ntpd: command not found 2024-09-22T03:36:04.452 INFO:teuthology.orchestra.run.smithi060.stdout:506 Cannot talk to daemon 2024-09-22T03:36:04.466 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:36:04.481 INFO:teuthology.orchestra.run.smithi060.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:36:04.550 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:36:04.553 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:36:04.553 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-22T03:36:04.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:36:04.553 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:36:04.554 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:36:04.554 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:36:04.554 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:36:04.562 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:36:04.562 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-22T03:36:04.562 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-22T03:36:04.563 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:36:04.568 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:36:04.568 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:36:04.568 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-22T03:36:04.568 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:36:04.711 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-22T03:36:04.711 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-22T03:36:04.887 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:36:04.888 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:36:04.888 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:36:04.915 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:36:04.915 DEBUG:teuthology.orchestra.run.smithi060:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:36:04.982 DEBUG:teuthology.orchestra.run.smithi060:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:36:05.060 DEBUG:teuthology.orchestra.run.smithi060:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:36:05.125 INFO:teuthology.orchestra.run.smithi060.stdout:check_obsoletes = 1 2024-09-22T03:36:05.126 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum clean all 2024-09-22T03:36:05.379 INFO:teuthology.orchestra.run.smithi060.stdout:45 files removed 2024-09-22T03:36:05.406 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-radosgw 2024-09-22T03:36:05.981 INFO:teuthology.orchestra.run.smithi060.stdout:ceph packages for x86_64 320 kB/s | 88 kB 00:00 2024-09-22T03:36:06.374 INFO:teuthology.orchestra.run.smithi060.stdout:ceph noarch packages 70 kB/s | 24 kB 00:00 2024-09-22T03:36:06.691 INFO:teuthology.orchestra.run.smithi060.stdout:ceph source packages 7.4 kB/s | 2.1 kB 00:00 2024-09-22T03:36:08.249 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.3 MB 00:01 2024-09-22T03:36:10.017 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-22T03:36:14.410 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-22T03:36:16.252 INFO:teuthology.orchestra.run.smithi060.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 19 kB 00:00 2024-09-22T03:36:18.590 INFO:teuthology.orchestra.run.smithi060.stdout:Extra Packages for Enterprise Linux 10 MB/s | 23 MB 00:02 2024-09-22T03:36:24.187 INFO:teuthology.orchestra.run.smithi060.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T03:36:26.510 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:36:26.528 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrading: 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-22T03:36:26.529 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:36:26.530 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:36:26.531 INFO:teuthology.orchestra.run.smithi060.stdout:Installing weak dependencies: 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Install 31 Packages 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Upgrade 2 Packages 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 60 M 2024-09-22T03:36:26.532 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:36:27.305 INFO:teuthology.orchestra.run.smithi060.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 12 MB/s | 6.1 MB 00:00 2024-09-22T03:36:27.355 INFO:teuthology.orchestra.run.smithi060.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 505 kB/s | 25 kB 00:00 2024-09-22T03:36:27.448 INFO:teuthology.orchestra.run.smithi060.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 8.6 MB/s | 808 kB 00:00 2024-09-22T03:36:27.548 INFO:teuthology.orchestra.run.smithi060.stdout:(4/33): libradosstriper1-19.3.0-5092.g62da54b4. 5.5 MB/s | 558 kB 00:00 2024-09-22T03:36:27.823 INFO:teuthology.orchestra.run.smithi060.stdout:(5/33): ceph-common-19.3.0-5092.g62da54b4.el9.x 19 MB/s | 20 MB 00:01 2024-09-22T03:36:27.948 INFO:teuthology.orchestra.run.smithi060.stdout:(6/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 8.8 MB/s | 10 MB 00:01 2024-09-22T03:36:27.973 INFO:teuthology.orchestra.run.smithi060.stdout:(7/33): python3-ceph-argparse-19.3.0-5092.g62da 300 kB/s | 45 kB 00:00 2024-09-22T03:36:27.998 INFO:teuthology.orchestra.run.smithi060.stdout:(8/33): python3-ceph-common-19.3.0-5092.g62da54 2.9 MB/s | 148 kB 00:00 2024-09-22T03:36:28.024 INFO:teuthology.orchestra.run.smithi060.stdout:(9/33): python3-cephfs-19.3.0-5092.g62da54b4.el 3.1 MB/s | 159 kB 00:00 2024-09-22T03:36:28.058 INFO:teuthology.orchestra.run.smithi060.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 5.2 MB/s | 314 kB 00:00 2024-09-22T03:36:28.083 INFO:teuthology.orchestra.run.smithi060.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-22T03:36:28.108 INFO:teuthology.orchestra.run.smithi060.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.9 MB/s | 99 kB 00:00 2024-09-22T03:36:28.192 INFO:teuthology.orchestra.run.smithi060.stdout:(13/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_ 8.3 MB/s | 5.4 MB 00:00 2024-09-22T03:36:28.284 INFO:teuthology.orchestra.run.smithi060.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 533 kB/s | 107 kB 00:00 2024-09-22T03:36:28.326 INFO:teuthology.orchestra.run.smithi060.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-09-22T03:36:28.384 INFO:teuthology.orchestra.run.smithi060.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 835 kB/s | 160 kB 00:00 2024-09-22T03:36:28.410 INFO:teuthology.orchestra.run.smithi060.stdout:(17/33): libnbd-1.20.2-2.el9.x86_64.rpm 566 kB/s | 171 kB 00:00 2024-09-22T03:36:28.468 INFO:teuthology.orchestra.run.smithi060.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-09-22T03:36:28.494 INFO:teuthology.orchestra.run.smithi060.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 504 kB/s | 42 kB 00:00 2024-09-22T03:36:28.527 INFO:teuthology.orchestra.run.smithi060.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-22T03:36:28.552 INFO:teuthology.orchestra.run.smithi060.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-22T03:36:28.578 INFO:teuthology.orchestra.run.smithi060.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-09-22T03:36:28.612 INFO:teuthology.orchestra.run.smithi060.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-09-22T03:36:28.678 INFO:teuthology.orchestra.run.smithi060.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 460 kB/s | 26 kB 00:00 2024-09-22T03:36:28.704 INFO:teuthology.orchestra.run.smithi060.stdout:(25/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2024-09-22T03:36:28.729 INFO:teuthology.orchestra.run.smithi060.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-22T03:36:28.771 INFO:teuthology.orchestra.run.smithi060.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-22T03:36:28.830 INFO:teuthology.orchestra.run.smithi060.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-22T03:36:28.938 INFO:teuthology.orchestra.run.smithi060.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-09-22T03:36:29.055 INFO:teuthology.orchestra.run.smithi060.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-09-22T03:36:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.9 MB/s | 4.4 MB 00:00 2024-09-22T03:36:29.206 INFO:teuthology.orchestra.run.smithi060.stdout:(32/33): librados2-19.3.0-5092.g62da54b4.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-22T03:36:29.472 INFO:teuthology.orchestra.run.smithi060.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 9.1 MB/s | 3.8 MB 00:00 2024-09-22T03:36:29.478 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:36:29.478 INFO:teuthology.orchestra.run.smithi060.stdout:Total 20 MB/s | 60 MB 00:02 2024-09-22T03:36:29.603 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:36:29.645 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:36:29.645 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:36:30.349 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:36:30.350 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:36:31.969 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:36:32.518 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-22T03:36:32.566 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:36:32.659 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:36:32.759 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:36:32.784 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:36:32.831 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:36:32.877 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-22T03:36:32.905 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-22T03:36:32.931 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-22T03:36:32.953 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:36:32.979 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:36:33.173 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:36:33.203 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:36:33.255 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-22T03:36:33.299 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-22T03:36:33.328 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:36:33.418 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-22T03:36:33.441 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:36:33.529 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:36:33.574 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-22T03:36:33.685 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-22T03:36:33.717 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-22T03:36:33.763 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-22T03:36:33.798 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-22T03:36:33.828 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-22T03:36:33.863 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-22T03:36:34.242 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-22T03:36:34.289 INFO:teuthology.orchestra.run.smithi060.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:36:34.365 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:36:34.394 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-22T03:36:34.431 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-22T03:36:35.044 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-22T03:36:35.167 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:36:35.239 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:36:35.325 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-22T03:36:35.353 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-22T03:36:37.940 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:36:37.982 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:36:38.718 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:36:38.761 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:36:38.856 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:36:38.856 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:36:38.856 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:36:38.857 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:38.896 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /sys 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /proc 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /mnt 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /var/tmp 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /home 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /root 2024-09-22T03:36:49.756 INFO:teuthology.orchestra.run.smithi060.stdout:skipping the directory /tmp 2024-09-22T03:36:49.757 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:50.913 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:36:50.940 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:36:50.940 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:36:50.940 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:36:50.940 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:36:50.941 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:36:50.941 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:50.943 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:36:51.032 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:36:51.032 INFO:teuthology.orchestra.run.smithi060.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:36:52.345 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-22T03:36:52.346 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-22T03:36:52.347 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-22T03:36:52.348 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-22T03:36:52.348 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:36:52.348 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-22T03:36:52.348 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-22T03:36:53.238 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout:Upgraded: 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:36:53.239 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:36:53.240 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:53.241 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:36:53.554 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-test 2024-09-22T03:36:54.156 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:36:54.295 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:36:54.296 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:54.296 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:36:54.296 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:36:54.297 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:54.298 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:36:54.298 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:36:54.298 INFO:teuthology.orchestra.run.smithi060.stdout:Install 9 Packages 2024-09-22T03:36:54.298 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:36:54.298 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 83 M 2024-09-22T03:36:54.299 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 775 M 2024-09-22T03:36:54.299 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:36:55.054 INFO:teuthology.orchestra.run.smithi060.stdout:(1/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 809 kB/s | 175 kB 00:00 2024-09-22T03:36:55.296 INFO:teuthology.orchestra.run.smithi060.stdout:(2/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 205 kB/s | 94 kB 00:00 2024-09-22T03:36:55.497 INFO:teuthology.orchestra.run.smithi060.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 914 kB/s | 183 kB 00:00 2024-09-22T03:36:55.805 INFO:teuthology.orchestra.run.smithi060.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 3.3 MB/s | 1.0 MB 00:00 2024-09-22T03:36:55.830 INFO:teuthology.orchestra.run.smithi060.stdout:(5/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 662 kB/s | 513 kB 00:00 2024-09-22T03:36:55.922 INFO:teuthology.orchestra.run.smithi060.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-22T03:36:55.947 INFO:teuthology.orchestra.run.smithi060.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 544 kB/s | 64 kB 00:00 2024-09-22T03:36:56.015 INFO:teuthology.orchestra.run.smithi060.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.3 MB/s | 122 kB 00:00 2024-09-22T03:36:57.223 INFO:teuthology.orchestra.run.smithi060.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 34 MB/s | 81 MB 00:02 2024-09-22T03:36:57.226 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:36:57.226 INFO:teuthology.orchestra.run.smithi060.stdout:Total 28 MB/s | 83 MB 00:02 2024-09-22T03:36:57.313 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:36:57.339 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:36:57.339 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:36:57.966 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:36:57.966 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:36:58.885 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:36:58.962 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-22T03:36:59.020 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-22T03:36:59.078 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-22T03:36:59.105 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-22T03:36:59.136 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:36:59.168 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:36:59.239 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:36:59.278 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-22T03:36:59.304 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:37:10.778 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:37:10.872 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:37:16.945 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-22T03:37:16.946 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-22T03:37:17.376 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-22T03:37:17.376 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:17.376 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:37:17.376 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:17.377 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:17.724 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph 2024-09-22T03:37:18.321 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:00:54 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:18.469 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:37:18.471 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:37:18.472 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:37:18.473 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:37:18.474 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:18.475 INFO:teuthology.orchestra.run.smithi060.stdout:Install 37 Packages 2024-09-22T03:37:18.475 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:18.475 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 84 M 2024-09-22T03:37:18.475 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 896 M 2024-09-22T03:37:18.475 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:37:19.132 INFO:teuthology.orchestra.run.smithi060.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-22T03:37:19.315 INFO:teuthology.orchestra.run.smithi060.stdout:(2/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 5.3 MB/s | 1.6 MB 00:00 2024-09-22T03:37:19.365 INFO:teuthology.orchestra.run.smithi060.stdout:(3/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 6.2 MB/s | 2.2 MB 00:00 2024-09-22T03:37:19.466 INFO:teuthology.orchestra.run.smithi060.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 2.9 MB/s | 292 kB 00:00 2024-09-22T03:37:19.591 INFO:teuthology.orchestra.run.smithi060.stdout:(5/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-22T03:37:20.467 INFO:teuthology.orchestra.run.smithi060.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 253 kB/s | 253 kB 00:01 2024-09-22T03:37:21.325 INFO:teuthology.orchestra.run.smithi060.stdout:(7/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86_ 35 MB/s | 70 MB 00:02 2024-09-22T03:37:21.734 INFO:teuthology.orchestra.run.smithi060.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 84 kB/s | 106 kB 00:01 2024-09-22T03:37:21.951 INFO:teuthology.orchestra.run.smithi060.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 532 kB/s | 1.2 MB 00:02 2024-09-22T03:37:22.034 INFO:teuthology.orchestra.run.smithi060.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 191 kB/s | 135 kB 00:00 2024-09-22T03:37:22.168 INFO:teuthology.orchestra.run.smithi060.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-22T03:37:22.218 INFO:teuthology.orchestra.run.smithi060.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-22T03:37:22.260 INFO:teuthology.orchestra.run.smithi060.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 838 kB/s | 35 kB 00:00 2024-09-22T03:37:22.302 INFO:teuthology.orchestra.run.smithi060.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-22T03:37:22.344 INFO:teuthology.orchestra.run.smithi060.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-22T03:37:22.386 INFO:teuthology.orchestra.run.smithi060.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-22T03:37:22.461 INFO:teuthology.orchestra.run.smithi060.stdout:(17/37): python3-requests-2.25.1-8.el9.noarch.r 172 kB/s | 125 kB 00:00 2024-09-22T03:37:22.495 INFO:teuthology.orchestra.run.smithi060.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-09-22T03:37:22.545 INFO:teuthology.orchestra.run.smithi060.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-22T03:37:22.579 INFO:teuthology.orchestra.run.smithi060.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 532 kB/s | 43 kB 00:00 2024-09-22T03:37:22.604 INFO:teuthology.orchestra.run.smithi060.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.0 MB/s | 172 kB 00:00 2024-09-22T03:37:22.629 INFO:teuthology.orchestra.run.smithi060.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-22T03:37:22.654 INFO:teuthology.orchestra.run.smithi060.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 709 kB/s | 18 kB 00:00 2024-09-22T03:37:22.680 INFO:teuthology.orchestra.run.smithi060.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 929 kB/s | 23 kB 00:00 2024-09-22T03:37:22.705 INFO:teuthology.orchestra.run.smithi060.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-22T03:37:22.730 INFO:teuthology.orchestra.run.smithi060.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-22T03:37:22.755 INFO:teuthology.orchestra.run.smithi060.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-22T03:37:22.781 INFO:teuthology.orchestra.run.smithi060.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-22T03:37:22.806 INFO:teuthology.orchestra.run.smithi060.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-22T03:37:22.831 INFO:teuthology.orchestra.run.smithi060.stdout:(30/37): python3-urllib3-1.26.5-6.el9.noarch.rp 246 kB/s | 215 kB 00:00 2024-09-22T03:37:22.848 INFO:teuthology.orchestra.run.smithi060.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 404 kB/s | 16 kB 00:00 2024-09-22T03:37:22.882 INFO:teuthology.orchestra.run.smithi060.stdout:(32/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-09-22T03:37:22.907 INFO:teuthology.orchestra.run.smithi060.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-22T03:37:22.933 INFO:teuthology.orchestra.run.smithi060.stdout:(34/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 891 kB/s | 90 kB 00:00 2024-09-22T03:37:22.966 INFO:teuthology.orchestra.run.smithi060.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-22T03:37:22.983 INFO:teuthology.orchestra.run.smithi060.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-22T03:37:23.025 INFO:teuthology.orchestra.run.smithi060.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-22T03:37:23.031 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:37:23.031 INFO:teuthology.orchestra.run.smithi060.stdout:Total 18 MB/s | 84 MB 00:04 2024-09-22T03:37:23.184 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:37:23.231 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:37:23.231 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:37:24.071 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:37:24.071 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:37:25.786 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:37:25.844 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-22T03:37:25.878 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-22T03:37:25.905 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-22T03:37:25.982 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-22T03:37:26.021 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-22T03:37:26.054 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-22T03:37:26.150 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-22T03:37:26.226 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-22T03:37:26.282 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-22T03:37:26.316 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-22T03:37:26.346 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-22T03:37:26.380 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-22T03:37:26.431 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-22T03:37:26.473 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-22T03:37:26.516 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-22T03:37:26.549 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-22T03:37:26.599 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-22T03:37:26.639 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-22T03:37:26.716 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-22T03:37:26.867 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-22T03:37:26.949 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-22T03:37:27.008 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-22T03:37:27.046 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-22T03:37:27.104 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-22T03:37:27.185 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-22T03:37:27.390 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-22T03:37:27.442 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-22T03:37:27.523 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-22T03:37:27.612 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-22T03:37:27.660 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-22T03:37:27.802 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-22T03:37:28.004 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-22T03:37:28.034 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:37:28.062 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:39.303 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:37:39.332 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:39.912 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:37:39.938 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:37:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:37:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:37:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:37:39.939 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:37:39.939 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:40.188 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:37:40.218 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:37:40.219 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:37:40.219 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:37:40.219 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:37:40.219 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:37:40.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:40.288 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-22T03:37:47.065 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-22T03:37:47.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-22T03:37:47.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-22T03:37:47.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-22T03:37:47.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-22T03:37:47.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-22T03:37:47.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-22T03:37:47.068 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:48.194 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:37:48.195 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:37:48.196 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:48.197 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:48.494 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-base 2024-09-22T03:37:49.101 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:25 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:49.193 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:37:49.237 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:49.239 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:37:49.239 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:49.303 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install cephadm 2024-09-22T03:37:49.911 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:25 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:50.049 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:37:50.050 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:50.051 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:37:50.051 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:50.051 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 784 k 2024-09-22T03:37:50.051 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 791 k 2024-09-22T03:37:50.051 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:37:50.329 INFO:teuthology.orchestra.run.smithi060.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-22T03:37:50.330 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:37:50.330 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-22T03:37:50.330 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:37:50.336 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:37:50.336 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:37:50.342 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:37:50.342 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:37:50.414 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:37:50.908 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:37:51.002 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:37:51.377 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:37:51.631 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:37:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:51.632 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:37:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:37:51.632 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:51.632 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:51.867 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:37:52.470 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:28 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:52.607 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:52.607 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:52.607 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-22T03:37:52.607 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:52.607 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 158 k 2024-09-22T03:37:52.608 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 644 k 2024-09-22T03:37:52.609 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:37:52.810 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 786 kB/s | 158 kB 00:00 2024-09-22T03:37:52.811 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:37:52.811 INFO:teuthology.orchestra.run.smithi060.stdout:Total 778 kB/s | 158 kB 00:00 2024-09-22T03:37:52.811 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:37:52.822 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:37:52.822 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:37:52.882 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:37:52.882 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:37:53.039 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:37:53.127 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:37:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:37:53.155 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:37:53.155 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:37:53.155 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:37:54.096 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:54.097 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:37:54.097 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:37:54.097 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:54.097 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:54.319 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr 2024-09-22T03:37:54.930 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:30 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:55.022 INFO:teuthology.orchestra.run.smithi060.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:37:55.067 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:55.068 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:37:55.068 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:37:55.135 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:37:55.737 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:31 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:37:55.875 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:37:55.876 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:55.876 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:37:55.877 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout:Install 11 Packages 2024-09-22T03:37:55.878 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:37:55.879 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 18 M 2024-09-22T03:37:55.879 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 156 M 2024-09-22T03:37:55.879 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:37:56.430 INFO:teuthology.orchestra.run.smithi060.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 240 kB/s | 34 kB 00:00 2024-09-22T03:37:56.455 INFO:teuthology.orchestra.run.smithi060.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 98 kB/s | 16 kB 00:00 2024-09-22T03:37:56.572 INFO:teuthology.orchestra.run.smithi060.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-22T03:37:56.614 INFO:teuthology.orchestra.run.smithi060.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-09-22T03:37:56.665 INFO:teuthology.orchestra.run.smithi060.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 406 kB/s | 19 kB 00:00 2024-09-22T03:37:56.706 INFO:teuthology.orchestra.run.smithi060.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.0 MB/s | 548 kB 00:00 2024-09-22T03:37:56.750 INFO:teuthology.orchestra.run.smithi060.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-22T03:37:56.774 INFO:teuthology.orchestra.run.smithi060.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-22T03:37:56.816 INFO:teuthology.orchestra.run.smithi060.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-22T03:37:56.957 INFO:teuthology.orchestra.run.smithi060.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.0 MB/s | 2.0 MB 00:00 2024-09-22T03:37:57.116 INFO:teuthology.orchestra.run.smithi060.stdout:(11/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 17 MB/s | 14 MB 00:00 2024-09-22T03:37:57.120 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:37:57.120 INFO:teuthology.orchestra.run.smithi060.stdout:Total 15 MB/s | 18 MB 00:01 2024-09-22T03:37:57.152 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:37:57.173 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:37:57.173 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:37:57.448 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:37:57.448 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:37:58.024 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:37:58.115 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:37:58.153 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:37:58.256 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:37:58.392 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:37:58.440 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:37:58.526 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:37:58.575 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:37:58.603 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:37:58.636 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-22T03:38:02.006 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-22T03:38:02.056 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:38:03.755 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:04.072 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:04.073 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:04.318 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:38:04.919 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:40 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:05.057 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:05.058 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-09-22T03:38:05.058 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:38:05.058 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-09-22T03:38:05.058 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:38:05.059 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout:======================================================================================= 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout:Install 12 Packages 2024-09-22T03:38:05.060 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:05.061 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 38 M 2024-09-22T03:38:05.061 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 212 M 2024-09-22T03:38:05.061 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:05.497 INFO:teuthology.orchestra.run.smithi060.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.5 MB/s | 191 kB 00:00 2024-09-22T03:38:05.538 INFO:teuthology.orchestra.run.smithi060.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.7 MB/s | 801 kB 00:00 2024-09-22T03:38:05.589 INFO:teuthology.orchestra.run.smithi060.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-09-22T03:38:05.648 INFO:teuthology.orchestra.run.smithi060.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 307 kB/s | 18 kB 00:00 2024-09-22T03:38:05.689 INFO:teuthology.orchestra.run.smithi060.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 944 kB/s | 38 kB 00:00 2024-09-22T03:38:06.307 INFO:teuthology.orchestra.run.smithi060.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-22T03:38:06.398 INFO:teuthology.orchestra.run.smithi060.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-09-22T03:38:06.499 INFO:teuthology.orchestra.run.smithi060.stdout:(8/12): ceph-mgr-diskprediction-local-19.3.0-50 6.6 MB/s | 7.4 MB 00:01 2024-09-22T03:38:06.624 INFO:teuthology.orchestra.run.smithi060.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-09-22T03:38:06.866 INFO:teuthology.orchestra.run.smithi060.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.2 MB/s | 4.9 MB 00:01 2024-09-22T03:38:07.724 INFO:teuthology.orchestra.run.smithi060.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-22T03:38:10.766 INFO:teuthology.orchestra.run.smithi060.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-22T03:38:10.767 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:10.768 INFO:teuthology.orchestra.run.smithi060.stdout:Total 6.7 MB/s | 38 MB 00:05 2024-09-22T03:38:11.140 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:11.167 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:11.168 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:11.620 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:11.620 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:12.407 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:12.500 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:38:12.540 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:38:12.643 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:38:12.683 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:38:12.865 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:38:12.906 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:38:12.984 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:38:13.668 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:38:13.789 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:38:15.192 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:38:16.274 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:38:16.397 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:38:19.205 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:38:19.206 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:38:19.640 INFO:teuthology.orchestra.run.smithi060.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:19.641 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:20.025 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:38:20.638 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:01:56 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:20.778 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:38:20.779 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout:Install 13 Packages 2024-09-22T03:38:20.780 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:20.781 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.2 M 2024-09-22T03:38:20.781 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 26 M 2024-09-22T03:38:20.781 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:21.030 INFO:teuthology.orchestra.run.smithi060.stdout:(1/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 306 kB/s | 49 kB 00:00 2024-09-22T03:38:21.239 INFO:teuthology.orchestra.run.smithi060.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 52 kB/s | 19 kB 00:00 2024-09-22T03:38:21.256 INFO:teuthology.orchestra.run.smithi060.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 66 kB/s | 26 kB 00:00 2024-09-22T03:38:21.656 INFO:teuthology.orchestra.run.smithi060.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 355 kB/s | 222 kB 00:00 2024-09-22T03:38:21.731 INFO:teuthology.orchestra.run.smithi060.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 322 kB/s | 159 kB 00:00 2024-09-22T03:38:21.765 INFO:teuthology.orchestra.run.smithi060.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 548 kB/s | 279 kB 00:00 2024-09-22T03:38:21.799 INFO:teuthology.orchestra.run.smithi060.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-22T03:38:21.824 INFO:teuthology.orchestra.run.smithi060.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-22T03:38:21.866 INFO:teuthology.orchestra.run.smithi060.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.3 MB/s | 220 kB 00:00 2024-09-22T03:38:21.899 INFO:teuthology.orchestra.run.smithi060.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-22T03:38:21.933 INFO:teuthology.orchestra.run.smithi060.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-09-22T03:38:21.967 INFO:teuthology.orchestra.run.smithi060.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-22T03:38:22.000 INFO:teuthology.orchestra.run.smithi060.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 156 kB/s | 54 kB 00:00 2024-09-22T03:38:22.003 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:22.003 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-22T03:38:22.077 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:22.094 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:22.094 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:22.194 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:22.195 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:22.723 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:22.902 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:38:22.953 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:38:23.018 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:38:23.053 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:38:23.095 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:38:23.201 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:38:23.312 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:38:23.363 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:38:24.126 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:38:24.185 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:38:24.219 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:38:24.268 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:38:24.296 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:38:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:38:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-22T03:38:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:38:25.066 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:38:25.067 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:38:25.412 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:25.413 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:25.668 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:38:26.277 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:02 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:26.416 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:26.416 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:26.417 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:26.418 INFO:teuthology.orchestra.run.smithi060.stdout:Install 6 Packages 2024-09-22T03:38:26.418 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:26.418 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 7.0 M 2024-09-22T03:38:26.418 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 33 M 2024-09-22T03:38:26.418 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:27.044 INFO:teuthology.orchestra.run.smithi060.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-09-22T03:38:27.069 INFO:teuthology.orchestra.run.smithi060.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 670 kB/s | 154 kB 00:00 2024-09-22T03:38:27.127 INFO:teuthology.orchestra.run.smithi060.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 991 kB/s | 58 kB 00:00 2024-09-22T03:38:27.161 INFO:teuthology.orchestra.run.smithi060.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-22T03:38:27.194 INFO:teuthology.orchestra.run.smithi060.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-22T03:38:27.887 INFO:teuthology.orchestra.run.smithi060.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.7 MB/s | 6.0 MB 00:01 2024-09-22T03:38:27.889 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:27.890 INFO:teuthology.orchestra.run.smithi060.stdout:Total 4.8 MB/s | 7.0 MB 00:01 2024-09-22T03:38:27.964 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:27.976 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:27.977 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:28.084 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:28.084 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:28.403 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:28.577 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:38:28.639 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:38:29.202 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:38:29.294 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:38:29.370 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:38:29.426 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:38:30.194 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:38:30.467 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:30.468 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:30.756 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-fuse 2024-09-22T03:38:31.364 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:07 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:31.501 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:31.502 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:31.503 INFO:teuthology.orchestra.run.smithi060.stdout:Install 2 Packages 2024-09-22T03:38:31.503 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:31.503 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 963 k 2024-09-22T03:38:31.503 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 4.1 M 2024-09-22T03:38:31.503 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:31.705 INFO:teuthology.orchestra.run.smithi060.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 785 kB/s | 80 kB 00:00 2024-09-22T03:38:31.880 INFO:teuthology.orchestra.run.smithi060.stdout:(2/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.1 MB/s | 883 kB 00:00 2024-09-22T03:38:31.881 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:31.881 INFO:teuthology.orchestra.run.smithi060.stdout:Total 2.5 MB/s | 963 kB 00:00 2024-09-22T03:38:31.893 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:31.919 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:31.919 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:32.003 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:32.004 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:32.202 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:32.355 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:38:32.441 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:38:33.178 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:38:33.178 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-22T03:38:33.471 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:38:33.471 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:33.471 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:33.471 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:38:33.471 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:33.472 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:33.752 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install ceph-volume 2024-09-22T03:38:34.361 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:10 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:34.499 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:34.500 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:38:34.500 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 285 k 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 1.3 M 2024-09-22T03:38:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:35.076 INFO:teuthology.orchestra.run.smithi060.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 496 kB/s | 285 kB 00:00 2024-09-22T03:38:35.077 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:35.077 INFO:teuthology.orchestra.run.smithi060.stdout:Total 494 kB/s | 285 kB 00:00 2024-09-22T03:38:35.077 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:35.088 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:35.088 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:35.123 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:35.123 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:35.362 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:35.436 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:38:35.453 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:38:35.454 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:38:35.454 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:38:35.454 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:36.500 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:36.817 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados-devel 2024-09-22T03:38:37.426 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:13 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:37.566 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-22T03:38:37.567 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 125 k 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 445 k 2024-09-22T03:38:37.568 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:37.781 INFO:teuthology.orchestra.run.smithi060.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 586 kB/s | 125 kB 00:00 2024-09-22T03:38:37.781 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:37.782 INFO:teuthology.orchestra.run.smithi060.stdout:Total 582 kB/s | 125 kB 00:00 2024-09-22T03:38:37.782 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:37.788 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:37.788 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:37.856 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:37.856 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:38.016 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:38.124 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:38.581 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:38.859 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:39.101 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs2 2024-09-22T03:38:39.713 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:15 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:39.805 INFO:teuthology.orchestra.run.smithi060.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:39.851 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:39.853 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:39.853 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:39.922 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install libcephfs-devel 2024-09-22T03:38:40.526 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:40.666 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 32 k 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 142 k 2024-09-22T03:38:40.667 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:40.818 INFO:teuthology.orchestra.run.smithi060.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 211 kB/s | 32 kB 00:00 2024-09-22T03:38:40.818 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:40.819 INFO:teuthology.orchestra.run.smithi060.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-22T03:38:40.819 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:40.822 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:40.822 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:40.841 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:40.842 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:40.945 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:41.052 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:41.454 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:41.682 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:41.885 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librados2 2024-09-22T03:38:42.492 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:18 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:42.586 INFO:teuthology.orchestra.run.smithi060.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:42.633 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:42.634 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:42.635 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:42.704 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install librbd1 2024-09-22T03:38:43.316 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:19 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:43.408 INFO:teuthology.orchestra.run.smithi060.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:43.455 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:43.456 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:43.456 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:43.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rados 2024-09-22T03:38:44.134 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:20 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:44.227 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:44.273 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:44.274 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:44.274 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:44.345 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rgw 2024-09-22T03:38:44.951 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:20 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:45.045 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:45.091 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:45.092 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:45.093 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:45.162 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-cephfs 2024-09-22T03:38:45.770 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:21 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:45.862 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:45.909 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:45.910 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:45.911 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:45.978 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-rbd 2024-09-22T03:38:46.594 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:46.688 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:38:46.735 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:46.736 INFO:teuthology.orchestra.run.smithi060.stdout:Nothing to do. 2024-09-22T03:38:46.737 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:46.802 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-fuse 2024-09-22T03:38:47.412 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:23 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:47.550 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:47.550 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:47.551 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:47.552 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 91 k 2024-09-22T03:38:47.552 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 321 k 2024-09-22T03:38:47.552 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:47.746 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 469 kB/s | 91 kB 00:00 2024-09-22T03:38:47.747 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:47.747 INFO:teuthology.orchestra.run.smithi060.stdout:Total 464 kB/s | 91 kB 00:00 2024-09-22T03:38:47.747 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:47.758 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:47.758 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:47.825 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:47.825 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:47.978 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:48.100 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:48.501 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:48.766 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:49.015 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-mirror 2024-09-22T03:38:49.621 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:25 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:49.763 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repo Size 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:49.787 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:49.788 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:49.788 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:49.788 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 3.5 M 2024-09-22T03:38:49.788 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 30 M 2024-09-22T03:38:49.788 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:50.108 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-22T03:38:50.109 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:50.109 INFO:teuthology.orchestra.run.smithi060.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-22T03:38:50.109 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:50.122 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:50.122 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:50.211 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:50.212 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:50.832 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:50.900 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:50.927 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:50.927 INFO:teuthology.orchestra.run.smithi060.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:38:50.928 INFO:teuthology.orchestra.run.smithi060.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:38:50.928 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:38:50.928 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:38:50.928 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:52.190 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:52.191 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:38:52.191 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:52.191 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:52.463 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install rbd-nbd 2024-09-22T03:38:53.083 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:02:29 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:38:53.222 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:38:53.223 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:53.223 INFO:teuthology.orchestra.run.smithi060.stdout: Package Architecture Version Repository Size 2024-09-22T03:38:53.223 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:53.223 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout:Install 1 Package 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:53.224 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 176 k 2024-09-22T03:38:53.225 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 685 k 2024-09-22T03:38:53.225 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:38:53.428 INFO:teuthology.orchestra.run.smithi060.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 863 kB/s | 176 kB 00:00 2024-09-22T03:38:53.430 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:53.430 INFO:teuthology.orchestra.run.smithi060.stdout:Total 856 kB/s | 176 kB 00:00 2024-09-22T03:38:53.430 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:38:53.440 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:38:53.441 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:38:53.509 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:38:53.510 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:38:53.671 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:38:53.779 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:54.231 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:54.498 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:38:54.776 DEBUG:teuthology.parallel:result is None 2024-09-22T03:38:54.777 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:38:54.922 DEBUG:teuthology.orchestra.run.smithi060:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:38:54.945 INFO:teuthology.orchestra.run.smithi060.stdout:19.3.0-5092.g62da54b4.el9 2024-09-22T03:38:54.945 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-22T03:38:54.945 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-22T03:38:54.947 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:38:54.947 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:54.947 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:38:55.012 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:38:55.012 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:55.012 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:38:55.078 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:38:55.144 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:38:55.144 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:55.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:38:55.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:38:55.275 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:38:55.275 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:55.275 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:38:55.340 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:38:55.406 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:38:55.487 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:38:55.487 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 777 /var/log/ceph 2024-09-22T03:38:55.529 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:38:55.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:38:55.595 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:38:55.596 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:38:55.668 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:38:55.669 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:38:55.669 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_write}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914359', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi060.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBClHYyBnZYLT34NhrwRaTFI7aZmDWB66VjtrXRLtFRaCIMtJtKxvIYxOlKSKyPbR4puycqxsA1yzZDF4XrVKVrA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:38:55.669 DEBUG:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:38:55.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:38:55.783 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:55.783 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:38:55.839 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:38:55.839 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_1 2024-09-22T03:38:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:38:55.894 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-22 03:38:51.176881211 +0000 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-22 03:35:45.591389983 +0000 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-22 03:35:45.591389983 +0000 2024-09-22T03:38:55.895 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-22 03:35:45.591389983 +0000 2024-09-22T03:38:55.895 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:38:55.958 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-22T03:38:55.959 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-22T03:38:55.959 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.6302e-05 s, 5.3 MB/s 2024-09-22T03:38:55.960 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:38:56.017 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_2 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-22 03:38:51.177881201 +0000 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-22 03:35:45.919386589 +0000 2024-09-22T03:38:56.073 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-22 03:35:45.919386589 +0000 2024-09-22T03:38:56.074 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-22 03:35:45.919386589 +0000 2024-09-22T03:38:56.074 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:38:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-22T03:38:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-22T03:38:56.136 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 9.9204e-05 s, 5.2 MB/s 2024-09-22T03:38:56.138 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:38:56.194 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_3 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-22 03:38:51.177881201 +0000 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-22 03:35:46.244383227 +0000 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-22 03:35:46.244383227 +0000 2024-09-22T03:38:56.251 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-22 03:35:46.244383227 +0000 2024-09-22T03:38:56.251 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:38:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-22T03:38:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-22T03:38:56.314 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 0.000101629 s, 5.0 MB/s 2024-09-22T03:38:56.316 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:38:56.373 DEBUG:teuthology.orchestra.run.smithi060:> stat /dev/vg_nvme/lv_4 2024-09-22T03:38:56.428 INFO:teuthology.orchestra.run.smithi060.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Access: 2024-09-22 03:38:51.178881191 +0000 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Modify: 2024-09-22 03:35:46.571379843 +0000 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout:Change: 2024-09-22 03:35:46.571379843 +0000 2024-09-22T03:38:56.429 INFO:teuthology.orchestra.run.smithi060.stdout: Birth: 2024-09-22 03:35:46.571379843 +0000 2024-09-22T03:38:56.429 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:38:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records in 2024-09-22T03:38:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:1+0 records out 2024-09-22T03:38:56.493 INFO:teuthology.orchestra.run.smithi060.stderr:512 bytes copied, 8.6641e-05 s, 5.9 MB/s 2024-09-22T03:38:56.494 DEBUG:teuthology.orchestra.run.smithi060:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:38:56.550 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:38:56.551 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:38:56.551 INFO:tasks.ceph:Generating config... 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-22T03:38:56.552 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:38:56.553 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:38:56.553 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T03:38:56.553 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T03:38:56.553 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T03:38:56.554 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T03:38:56.555 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T03:38:56.556 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-22T03:38:56.557 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:38:56.558 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T03:38:56.559 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:38:56.559 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:38:56.559 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:38:56.559 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:38:56.559 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:38:56.559 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:38:56.560 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:38:56.667 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:38:56.671 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:38:56.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:38:56.747 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.60')] 2024-09-22T03:38:56.748 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.60', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:38:56.749 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:56.749 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:38:56.803 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.60 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:38:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:38:56.896+0000 7fb0ffefef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:38:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:38:56.896+0000 7fb0ffefef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:38:56.898 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:38:56.897+0000 7fb0ffefef00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: generated fsid e5df6476-8bd7-4110-9bf2-d453354cc705 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:setting min_mon_release = quincy 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:epoch 0 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:fsid e5df6476-8bd7-4110-9bf2-d453354cc705 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:last_changed 2024-09-22T03:38:56.898265+0000 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:created 2024-09-22T03:38:56.898265+0000 2024-09-22T03:38:56.899 INFO:teuthology.orchestra.run.smithi060.stdout:min_mon_release 17 (quincy) 2024-09-22T03:38:56.900 INFO:teuthology.orchestra.run.smithi060.stdout:election_strategy: 1 2024-09-22T03:38:56.900 INFO:teuthology.orchestra.run.smithi060.stdout:0: [v2:172.21.15.60:3300/0,v1:172.21.15.60:6789/0] mon.a 2024-09-22T03:38:56.900 INFO:teuthology.orchestra.run.smithi060.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:38:56.901 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:38:56.916 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e5df6476-8bd7-4110-9bf2-d453354cc705... 2024-09-22T03:38:56.917 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout:[global] 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: chdir = "" 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: auth supported = none 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: filestore xattr use omap = true 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: mon clock drift allowed = 1.000 2024-09-22T03:38:57.001 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: osd crush chooseleaf type = 0 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: auth debug = true 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on old message = true 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: ms die on bug = true 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: mon pg warn max object skew = 0 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:38:57.002 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default size = 2 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow primary affinity = true 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd allow pg remap = true 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on no sortbitwise = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon warn on too few osds = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: mon_allow_pool_size_one = true 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.003 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: osd default data pool replay window = 5 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: mon allow pool delete = true 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: mon cluster log file level = debug 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: debug asserts on shutdown = true 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: mon health detail to clog = false 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: mon host = 172.21.15.60 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: auth client required = none 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: auth cluster required = none 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: auth service required = none 2024-09-22T03:38:57.004 INFO:teuthology.orchestra.run.smithi060.stdout: debug asok = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug bdev = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluefs = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug bluestore = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug buffer = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug civetweb = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug client = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug compressor = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug context = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug crush = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug crypto = 0/0 2024-09-22T03:38:57.005 INFO:teuthology.orchestra.run.smithi060.stdout: debug dpdk = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug eventtrace = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug filer = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug finisher = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug fuse = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug heartbeatmap = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug javaclient = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug journaler = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug kstore = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug leveldb = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug lockdep = 0/0 2024-09-22T03:38:57.006 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds balancer = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds locker = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds log = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds log expire = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mds migrator = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug memdb = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgrc = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug objclass = 0/0 2024-09-22T03:38:57.007 INFO:teuthology.orchestra.run.smithi060.stdout: debug objectcacher = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug objecter = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug optracker = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug perfcounter = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rados = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd mirror = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rbd replay = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug refs = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw lifecycle = 0/0 2024-09-22T03:38:57.008 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw notification = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug rgw sync = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug rocksdb = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug striper = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug throttle = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug timer = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: debug tp = 0/0 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms client mode = crc 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms cluster mode = crc 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon client mode = crc 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon cluster mode = crc 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms mon service mode = crc 2024-09-22T03:38:57.009 INFO:teuthology.orchestra.run.smithi060.stdout: ms service mode = crc 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: fsid = e5df6476-8bd7-4110-9bf2-d453354cc705 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout:[osd] 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd journal size = 100 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub load threshold = 5.0 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd scrub max interval = 600 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd recover clone overlap = true 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd recovery max chunk = 1048576 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.010 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug shutdown = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug op order = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug verify stray on activate = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug trim objects = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd open classes on start = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug pg log writeout = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: osd map max advance = 10 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: journal zero on create = true 2024-09-22T03:38:57.011 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: filestore apply finisher threads = 3 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: bdev debug aio = true 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: osd debug misdirected ops = true 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug monc = 20 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: crimson osd obc lru size = 10 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug filestore = 0/0 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug journal = 0/0 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 0/0 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug osd = 0/0 2024-09-22T03:38:57.012 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore backref = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore cache = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore cleaner = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore device = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore epm = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore fixedkv tree = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore journal = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore lba = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore odata = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore ompap = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore onode = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore t = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: debug seastore tm = 20 2024-09-22T03:38:57.013 INFO:teuthology.orchestra.run.smithi060.stdout: osd objectstore = seastore 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: osd shutdown pgref assert = True 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr] 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 1 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: debug mgr = 20 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 20 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: mgr/telemetry/nag = false 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout:[mon] 2024-09-22T03:38:57.014 INFO:teuthology.orchestra.run.smithi060.stdout: debug ms = 0/0 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: debug mon = 0/0 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: debug paxos = 0/0 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: debug auth = 20 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon data avail warn = 5 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon mgr mkfs grace = 240 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd reporter subtree level = osd 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon osd prime pg temp = true 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:38:57.015 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: # 1m isn't quite enough 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: osd pool default crimson = True 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout:[client] 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: rgw cache enabled = true 2024-09-22T03:38:57.016 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable ops log = true 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: rgw enable usage log = true 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout:[mon.a] 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.3] 2024-09-22T03:38:57.017 INFO:teuthology.orchestra.run.smithi060.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:38:57.018 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:38:57.018 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:38:57.126 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:38:57.126 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:57.126 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:38:57.142 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:57.142 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:38:57.197 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:38:57.197 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:57.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:38:57.198 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:38:57.273 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:38:57.273 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:38:57.330 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:38:57.330 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:38:57.330 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:38:57.431 INFO:teuthology.orchestra.run.smithi060.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:38:57.434 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:38:57.434 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:38:57.434 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:38:57.486 INFO:teuthology.orchestra.run.smithi060.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:38:57.498 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:38:57.498 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi060.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:38:57.498 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:38:57.523 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:38:57.523 INFO:tasks.ceph:role: osd.0 2024-09-22T03:38:57.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:38:57.524 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:38:57.589 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:38:57.590 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:38:57.621 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-22T03:38:57.624 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-22T03:38:57.624 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:38:57.694 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:38:57.765 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:38:57.831 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:38:57.831 INFO:tasks.ceph:role: osd.1 2024-09-22T03:38:57.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:38:57.831 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:38:57.897 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:38:57.930 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-22T03:38:57.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-22T03:38:57.932 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:38:58.002 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:38:58.072 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:38:58.138 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:38:58.139 INFO:tasks.ceph:role: osd.2 2024-09-22T03:38:58.139 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:38:58.139 DEBUG:teuthology.orchestra.run.smithi060:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:38:58.203 INFO:teuthology.orchestra.run.smithi060.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:38:58.235 INFO:teuthology.orchestra.run.smithi060.stdout:Discarding blocks...Done. 2024-09-22T03:38:58.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi060.front.sepia.ceph.com -o noatime 2024-09-22T03:38:58.237 DEBUG:teuthology.orchestra.run.smithi060:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:38:58.305 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:38:58.375 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:38:58.698 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:38:58.699 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:38:58,699 seastar - Reactor backend: linux-aio 2024-09-22T03:38:58.711 INFO:teuthology.orchestra.run.smithi060.stderr:==27863==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:38:58.852 INFO:teuthology.orchestra.run.smithi060.stderr:DEBUG 2024-09-22 03:38:58,851 [shard 0:main] seastar - reactor::drain 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:38:59.384 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:38:59.447 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:38:59.451 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:38:59,451 seastar - Reactor backend: linux-aio 2024-09-22T03:38:59.461 INFO:teuthology.orchestra.run.smithi060.stderr:==27861==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:38:59.701 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xba17197 0xba17729 0xb9b1eb4 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x968cf8c 0x968d328 0x968d4ad 0x968d54f 0x968d668 0xbe289a9 0xbe28bb0 0xbe28c86 0xbe28dc1 0xbe291b5 0xbe28578 0xbe28658 0xbe2876c 0xbe2842f 0x94eb330 0x95d0372 0x9307833 0x9307a0d 0x9307b48 0x9307c02 0x9307cdc 0x9308228 0x9308500 0x93085b5 0x957fcfc 0x957ff61 0x95ae198 0x96b1946 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:38:59.702 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x15a449 0x4162a 0x41bbe 0x41c92 0x41f68 0x4458e 0xfe5da 0x5ca6522 0x5ca66fc 0x5ca69b3 0x5ca6aa8 0x961a19c 0xb9ff914 0xb9ffb30 0xb9ada78 0x9add401 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a1e7 0xfca1b 0x3869141 0xb836412 0xb83655a 0xb836643 0xb836903 0xb8369f6 0xb836ad6 0xb836d56 0xb837301 0xb83770c 0xb8379cb 0xb838488 0xba09156 0xb9ae311 0x9adcd51 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff90fbecb0 0xffffffff90fbf617 0xffffffff90fe47b3 0xffffffff90f8eb3e 0xffffffff90f9699b 0xffffffff90f96e1c 0xffffffff90c8c884 0xffffffff918d1db2 0xffffffff91a00bb2 2024-09-22T03:38:59.762 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.820 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d09 0xdd332 0x4ab1c 0xb837745 0xb8379cb 0xba09729 0xba0cf80 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.820 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42076 0x3ecea 0xfc9b1 0x387c2e3 0x387c67f 0x9adeeea 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.820 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.820 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.820 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a2ca 0xfd59e 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb837196 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xb9bdf19 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:38:59.821 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.878 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133caf 0xdd332 0x4b14c 0xb9cdba8 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:38:59.878 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.888 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 33 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c2b 0xdd332 0x4ab1c 0xb9cc4a3 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9b5fda 0xb9b6935 0xb9b6db7 0xb9b6fcc 0xb9cb233 0xb9cb472 0xb9a8391 0xb9a844a 0x9af5b0d 0x996290a 0x9962b26 0x99257c9 0x99259c6 0x94afcb3 0x94b4051 0x9257611 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.888 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.922 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d18 0xdd332 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9e0fae 0xba1c4c3 0xba1c84e 0xb9bcc48 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.922 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.946 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 91 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d09 0xdd332 0x4ab1c 0x3ac7302 0x3b8c153 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xba15051 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.946 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:38:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x390ff 0x4ab1c 0x9530d60 0x95c6c63 0x95c7175 0x9add047 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:38:59.980 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:00.020 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10d771 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:00.020 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525898 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:00.077 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:00.117 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525201 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:00.117 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:00.117 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c27 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:00.117 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:01.583 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:01.583 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:39:01.584 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:39:01.584 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:39:01.584 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:01.584 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:01.612 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:39:01.638 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:39:01.933 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:39:01.934 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:39:01,933 seastar - Reactor backend: linux-aio 2024-09-22T03:39:01.940 INFO:teuthology.orchestra.run.smithi060.stderr:==27910==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:39:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b568 0x10d76c 0xdd31a 0x4ab1c 0x601bacb 0x601bc69 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:02.048 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:02.082 INFO:teuthology.orchestra.run.smithi060.stderr:DEBUG 2024-09-22 03:39:02,082 [shard 0:main] seastar - reactor::drain 2024-09-22T03:39:02.613 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:02.614 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:39:02.614 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:39:02.614 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:39:02.614 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:02.614 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:02.672 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:39:02.677 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:39:02,677 seastar - Reactor backend: linux-aio 2024-09-22T03:39:02.686 INFO:teuthology.orchestra.run.smithi060.stderr:==27908==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:39:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4b12f 0x5fba850 0x5fc07f3 0x5fc09b5 0x601bbd4 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:02.751 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc926c 0xc9e39 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9d336b 0xb9d3620 0xb9bb9f1 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc9232 0xc9e39 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9d336b 0xb9d3620 0xb9bb9f1 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:02.987 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb836dd9 0xb837301 0xb83770c 0xb8379cb 0xb838488 0xba09156 0xb9ae311 0x9a94eb7 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8c125 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9e53f4 0xba14771 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x125011 0x44735 0xfb4fb 0x3767e9b 0x5ca664d 0x5ca69b3 0x5ca6aa8 0x961a19c 0xb9ff914 0xb9ffb30 0xb9ada78 0x9add401 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:03.045 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x117f96 0x3868e75 0x386922f 0x9826a54 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:03.103 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.144 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8bed2 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9d8b8b 0xb9d904e 0xba147dd 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.144 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.145 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc9277 0xc9e39 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9d16cf 0xb9d2155 0xba1750a 0xb9b1eb4 0x9826b31 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.145 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9add9dc 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.202 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x60f73e6 0x60f75d7 0x60f785f 0xba17baf 0xba17c96 0xba17ecc 0xba18068 0xba183fe 0xba18610 0xb9b548f 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.203 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.241 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:03.241 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.299 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55257b1 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:03.299 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.338 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253e6 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.339 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:03.340 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525dba 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:03.340 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:04.821 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:04.822 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:39:04.822 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:39:04.822 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:39:04.822 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:04.822 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:04.849 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:39:04.875 DEBUG:teuthology.orchestra.run.smithi060:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:39:05.166 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:39:05.167 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:39:05,167 seastar - Reactor backend: linux-aio 2024-09-22T03:39:05.172 INFO:teuthology.orchestra.run.smithi060.stderr:==27957==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:39:05.280 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc9232 0xc9e39 0x36074e8 0x362b837 0x362ba8c 0x5f93e3b 0x5fb583c 0x5fb5ad6 0x5f9c6fc 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:05.280 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:05.312 INFO:teuthology.orchestra.run.smithi060.stderr:DEBUG 2024-09-22 03:39:05,312 [shard 0:main] seastar - reactor::drain 2024-09-22T03:39:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:05.842 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:39:05.842 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:39:05.842 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:39:05.843 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:05.843 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:05.897 INFO:teuthology.orchestra.run.smithi060.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:39:05.901 INFO:teuthology.orchestra.run.smithi060.stderr:INFO 2024-09-22 03:39:05,901 seastar - Reactor backend: linux-aio 2024-09-22T03:39:05.911 INFO:teuthology.orchestra.run.smithi060.stderr:==27955==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:39:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156cf 0xdd312 0x4ab1c 0x5fccc96 0x5fcce5e 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:05.976 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x36109ab 0x3610d67 0xba0be37 0xba0c16c 0xba0c21c 0xba0c747 0xba0d1d1 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9ade1c4 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x968cf8c 0x968d328 0x968d4ad 0x968d54f 0x968d668 0xbe289a9 0xbe28bb0 0xbe28c86 0xbe28dc1 0xbe291b5 0xbe28578 0xbe28658 0xbe2876c 0xbe2842f 0x94eb330 0x95d0372 0x9307833 0x9307a0d 0x9307b48 0x9307c02 0x9307cdc 0x9308228 0x9308500 0x93085b5 0x957fcfc 0x957ff61 0x95ae198 0x96b1946 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:06.152 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.209 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42198 0x3ef7b 0xfd531 0x95c6055 0x95c6cac 0x95c7175 0x9adca6b 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.209 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff90f91862 0xffffffff90f9699b 0xffffffff90f96e1c 0xffffffff90c8c884 0xffffffff918d1db2 0xffffffff91a00bb2 2024-09-22T03:39:06.209 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab9d 0x5c73334 0xb9cda95 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.209 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x125011 0x421f3 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb836d56 0xb837301 0xb83770c 0xb8379cb 0xb838488 0xba09156 0xb9ae311 0x9a94487 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4b184 0x3b8bfdd 0x3b8c15e 0xb9d336b 0xb9d3620 0xb9bb9f1 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.267 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.268 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb8368b3 0xba0a005 0xba0a0f4 0xba0a374 0xba0a91f 0xba0ade1 0xba0b239 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:06.268 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9e850c 0xba170df 0xba17729 0xb9b22dd 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:06.326 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x53b10 0x4ab1c 0xb9aae29 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10796f 0xc9e1d 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.366 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3ef69 0xfd531 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb836d56 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xb9bdf19 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3779702 0x3b8c06d 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9d02c7 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.424 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253dd 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:39:06.463 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.559 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b568 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.559 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b558 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:39:06.560 INFO:teuthology.orchestra.run.smithi060.stderr:kernel callstack: 2024-09-22T03:39:08.024 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:Suppressions used: 2024-09-22T03:39:08.025 INFO:teuthology.orchestra.run.smithi060.stderr: count bytes template 2024-09-22T03:39:08.025 INFO:teuthology.orchestra.run.smithi060.stderr: 1 8 MallocExtension::Register 2024-09-22T03:39:08.025 INFO:teuthology.orchestra.run.smithi060.stderr:----------------------------------------------------- 2024-09-22T03:39:08.025 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:08.053 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:39:08.079 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:39:08.079 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:08.079 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:39:08.144 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:08.144 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:39:08.209 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:08.209 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:39:08.274 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:08.274 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:39:08.339 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:08.339 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:39:08.394 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:39:08.395 DEBUG:teuthology.orchestra.run.smithi060:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:39:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:[mgr.x] 2024-09-22T03:39:08.458 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBRke9mhQyuGRAAxU9fx3JSexBYf2cZx+qqAQ== 2024-09-22T03:39:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.0] 2024-09-22T03:39:08.458 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBTke9m5Je5JBAALBrkhfgpmyS8P/zfhvjGRw== 2024-09-22T03:39:08.458 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.1] 2024-09-22T03:39:08.459 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBWke9mY6lPMhAAqOdcg1EY78nt3uuF5caNTg== 2024-09-22T03:39:08.459 INFO:teuthology.orchestra.run.smithi060.stdout:[osd.2] 2024-09-22T03:39:08.459 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBake9mRpX+AxAA4IRiueo5brEITrWQ4VxxnA== 2024-09-22T03:39:08.459 INFO:teuthology.orchestra.run.smithi060.stdout:[client.0] 2024-09-22T03:39:08.459 INFO:teuthology.orchestra.run.smithi060.stdout: key = AQBRke9mgxDlHBAAGIT/VkqxkJc2sriryXa6Zg== 2024-09-22T03:39:08.460 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:39:08.562 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:39:08.623 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:39:08.686 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:39:08.748 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:39:08.813 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:39:08.814 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:39:08.839 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:39:08.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:08.948+0000 7f9b1f0ebc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:08.949 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:08.948+0000 7f9b1f0ebc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:08.963 DEBUG:teuthology.orchestra.run.smithi060:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:39:08.988 DEBUG:teuthology.orchestra.run.smithi060:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:39:09.043 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:39:09.043 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:39:09.044 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:39:09.085 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:39:09.085 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:39:09.085 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:39:09.086 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:39:09.088 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:39:09.088 DEBUG:tasks.ceph:set 0 configs 2024-09-22T03:39:09.088 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:39:09.088 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:39:09.174 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-22T03:39:09.170+0000 7f618b431c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.174 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-22T03:39:09.170+0000 7f618b431c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.174 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-22T03:39:09.171+0000 7f618b431c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.174 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-22T03:39:09.172+0000 7f0ce4eff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.211 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:09.204+0000 7f297ded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:09.212+0000 7f297ded1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.283 INFO:teuthology.orchestra.run.smithi060.stderr:adjusted tunables profile to default 2024-09-22T03:39:09.295 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:39:09.295 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:39:09.295 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:39:09.377 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-22T03:39:09.376+0000 7f0ce4eff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.378 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-22T03:39:09.376+0000 7f0ce4eff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.405 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:09.400+0000 7fbeef951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.408 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:09.407+0000 7fbeef951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr:2024-09-22T03:39:09.487+0000 7f6181df4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr:feature with important data. 2024-09-22T03:39:09.488 INFO:tasks.ceph.mon.a.smithi060.stderr: 2024-09-22T03:39:10.295 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:39:10.296 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:10.296 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:39:10.321 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:10.321 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:39:10.387 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:10.387 DEBUG:teuthology.orchestra.run.smithi060:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:39:10.452 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 93fabec7-09d0-47e9-b491-77ea0b9fed92 0 2024-09-22T03:39:10.604 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:10.597+0000 7fd2bea63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:10.607 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:10.606+0000 7fd2bea63640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:10.685 INFO:teuthology.orchestra.run.smithi060.stdout:0 2024-09-22T03:39:10.697 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 692744b2-4b3e-4cb0-b5ad-128748d4195e 1 2024-09-22T03:39:10.809 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:10.802+0000 7fcf58391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:10.814 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:10.810+0000 7fcf58391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:10.889 INFO:teuthology.orchestra.run.smithi060.stdout:1 2024-09-22T03:39:10.900 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd new 28a5418a-94c6-42c7-a358-feb8c06603d1 2 2024-09-22T03:39:11.012 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:11.005+0000 7f3d82b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:11.014 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:11.013+0000 7f3d82b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:11.092 INFO:teuthology.orchestra.run.smithi060.stdout:2 2024-09-22T03:39:11.103 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:39:11.133 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:39:11.133 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:39:11.175 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:39:11.175 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:39:11.205 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:39:11.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:39:11.248 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:39:11.248 DEBUG:teuthology.orchestra.run.smithi060:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:39:11.281 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:39:11.282 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:39:11.324 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:39:11.324 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:39:11.470 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:11.465+0000 7f2909b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:11.473 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:11.472+0000 7f2909b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:11.593 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:11.593 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":6,"fsid":"e5df6476-8bd7-4110-9bf2-d453354cc705","created":"2024-09-22T03:39:09.196669+0000","modified":"2024-09-22T03:39:11.091161+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:39:11.091161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"93fabec7-09d0-47e9-b491-77ea0b9fed92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"692744b2-4b3e-4cb0-b5ad-128748d4195e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28a5418a-94c6-42c7-a358-feb8c06603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:39:11.608 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:39:11.608 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:39:11.910 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:39:12.020 INFO:teuthology.misc.health.smithi060.stderr:2024-09-22T03:39:12.014+0000 7f3918efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:12.022 INFO:teuthology.misc.health.smithi060.stderr:2024-09-22T03:39:12.021+0000 7f3918efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:12.092 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-22T03:39:12.092 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":6,"fsid":"e5df6476-8bd7-4110-9bf2-d453354cc705","created":"2024-09-22T03:39:09.196669+0000","modified":"2024-09-22T03:39:11.091161+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:39:11.091161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"93fabec7-09d0-47e9-b491-77ea0b9fed92","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"692744b2-4b3e-4cb0-b5ad-128748d4195e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28a5418a-94c6-42c7-a358-feb8c06603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:39:12.103 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:39:15.002 INFO:tasks.ceph.mgr.x.smithi060.stderr:2024-09-22T03:39:15.001+0000 7f0c826a5640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:39:18.406 DEBUG:teuthology.orchestra.run.smithi060:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:39:18.519 INFO:teuthology.misc.health.smithi060.stderr:2024-09-22T03:39:18.514+0000 7ff3cf391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:18.522 INFO:teuthology.misc.health.smithi060.stderr:2024-09-22T03:39:18.521+0000 7ff3cf391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:18.726 INFO:teuthology.misc.health.smithi060.stdout: 2024-09-22T03:39:18.726 INFO:teuthology.misc.health.smithi060.stdout:{"epoch":10,"fsid":"e5df6476-8bd7-4110-9bf2-d453354cc705","created":"2024-09-22T03:39:09.196669+0000","modified":"2024-09-22T03:39:18.057924+0000","last_up_change":"2024-09-22T03:39:18.057924+0000","last_in_change":"2024-09-22T03:39:11.091161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:39:18.009385+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93fabec7-09d0-47e9-b491-77ea0b9fed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":206371290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":206371290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":206371290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":206371290}]},"public_addr":"172.21.15.60:6807/206371290","cluster_addr":"172.21.15.60:6806/206371290","heartbeat_back_addr":"172.21.15.60:6814/206371290","heartbeat_front_addr":"172.21.15.60:6815/206371290","state":["exists","up"]},{"osd":1,"uuid":"692744b2-4b3e-4cb0-b5ad-128748d4195e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":158194884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":158194884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":158194884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":158194884}]},"public_addr":"172.21.15.60:6803/158194884","cluster_addr":"172.21.15.60:6804/158194884","heartbeat_back_addr":"172.21.15.60:6802/158194884","heartbeat_front_addr":"172.21.15.60:6805/158194884","state":["exists","up"]},{"osd":2,"uuid":"28a5418a-94c6-42c7-a358-feb8c06603d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3746062838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3746062838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3746062838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":3746062838}]},"public_addr":"172.21.15.60:6808/3746062838","cluster_addr":"172.21.15.60:6809/3746062838","heartbeat_back_addr":"172.21.15.60:6810/3746062838","heartbeat_front_addr":"172.21.15.60:6811/3746062838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:39:18.739 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:39:18.739 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:39:18.740 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:39:18.863 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:18.853+0000 7fd6c68bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:18.865 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:18.864+0000 7fd6c68bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:20.066 INFO:teuthology.orchestra.run.smithi060.stderr:pool 'rbd' created 2024-09-22T03:39:20.090 DEBUG:teuthology.orchestra.run.smithi060:> rbd --cluster ceph pool init rbd 2024-09-22T03:39:20.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:20.210+0000 7f1e37640ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:20.213 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:20.210+0000 7f1e37640ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:20.217 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:20.215+0000 7f1e37640ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:23.191 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:39:23.192 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-22T03:39:23.192 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:39:23.310 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:23.303+0000 7f7c5d284640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:23.312 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:23.311+0000 7f7c5d284640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:23.515 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:23.528 INFO:teuthology.orchestra.run.smithi060.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-22T03:39:23.528 INFO:tasks.ceph_manager:config epoch is 1 2024-09-22T03:39:23.529 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:39:23.529 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:39:23.529 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:39:23.641 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:23.637+0000 7f38a9991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:23.643 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:23.642+0000 7f38a9991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:23.892 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:23.904 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6800","nonce":3755119619},{"type":"v1","addr":"172.21.15.60:6801","nonce":3755119619}]},"active_addr":"172.21.15.60:6801/3755119619","active_change":"2024-09-22T03:39:13.987038+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":1818714876}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2205141033}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":3188069949}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.60:0","nonce":2863022927}]}]} 2024-09-22T03:39:23.908 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:39:23.908 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:39:23.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:39:24.020 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.016+0000 7f7247c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.023 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.021+0000 7f7247c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.219 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:24.220 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"fsid":"e5df6476-8bd7-4110-9bf2-d453354cc705","created":"2024-09-22T03:39:09.196669+0000","modified":"2024-09-22T03:39:23.106412+0000","last_up_change":"2024-09-22T03:39:18.057924+0000","last_in_change":"2024-09-22T03:39:11.091161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:39:18.009385+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:39:19.083696+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93fabec7-09d0-47e9-b491-77ea0b9fed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":206371290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":206371290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":206371290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":206371290}]},"public_addr":"172.21.15.60:6807/206371290","cluster_addr":"172.21.15.60:6806/206371290","heartbeat_back_addr":"172.21.15.60:6814/206371290","heartbeat_front_addr":"172.21.15.60:6815/206371290","state":["exists","up"]},{"osd":1,"uuid":"692744b2-4b3e-4cb0-b5ad-128748d4195e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":158194884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":158194884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":158194884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":158194884}]},"public_addr":"172.21.15.60:6803/158194884","cluster_addr":"172.21.15.60:6804/158194884","heartbeat_back_addr":"172.21.15.60:6802/158194884","heartbeat_front_addr":"172.21.15.60:6805/158194884","state":["exists","up"]},{"osd":2,"uuid":"28a5418a-94c6-42c7-a358-feb8c06603d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3746062838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3746062838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3746062838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":3746062838}]},"public_addr":"172.21.15.60:6808/3746062838","cluster_addr":"172.21.15.60:6809/3746062838","heartbeat_back_addr":"172.21.15.60:6810/3746062838","heartbeat_front_addr":"172.21.15.60:6811/3746062838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:39:24.232 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:39:24.232 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:39:24.346 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.342+0000 7fead2ec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.348 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.347+0000 7fead2ec8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.547 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:24.547 INFO:teuthology.orchestra.run.smithi060.stdout:{"epoch":15,"fsid":"e5df6476-8bd7-4110-9bf2-d453354cc705","created":"2024-09-22T03:39:09.196669+0000","modified":"2024-09-22T03:39:23.106412+0000","last_up_change":"2024-09-22T03:39:18.057924+0000","last_in_change":"2024-09-22T03:39:11.091161+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:39:18.009385+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:39:19.083696+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"93fabec7-09d0-47e9-b491-77ea0b9fed92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6807","nonce":206371290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6806","nonce":206371290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6814","nonce":206371290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6815","nonce":206371290}]},"public_addr":"172.21.15.60:6807/206371290","cluster_addr":"172.21.15.60:6806/206371290","heartbeat_back_addr":"172.21.15.60:6814/206371290","heartbeat_front_addr":"172.21.15.60:6815/206371290","state":["exists","up"]},{"osd":1,"uuid":"692744b2-4b3e-4cb0-b5ad-128748d4195e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6803","nonce":158194884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6804","nonce":158194884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6802","nonce":158194884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6805","nonce":158194884}]},"public_addr":"172.21.15.60:6803/158194884","cluster_addr":"172.21.15.60:6804/158194884","heartbeat_back_addr":"172.21.15.60:6802/158194884","heartbeat_front_addr":"172.21.15.60:6805/158194884","state":["exists","up"]},{"osd":2,"uuid":"28a5418a-94c6-42c7-a358-feb8c06603d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6808","nonce":3746062838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6809","nonce":3746062838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6810","nonce":3746062838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.60:6811","nonce":3746062838}]},"public_addr":"172.21.15.60:6808/3746062838","cluster_addr":"172.21.15.60:6809/3746062838","heartbeat_back_addr":"172.21.15.60:6810/3746062838","heartbeat_front_addr":"172.21.15.60:6811/3746062838","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:39:24.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:39:24.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:39:24.561 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:39:24.680 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.675+0000 7f438d191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.675+0000 7efc02191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.681 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.675+0000 7f7daef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.682+0000 7f438d191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.683 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.682+0000 7efc02191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.684 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.682+0000 7f7daef91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.716 INFO:teuthology.orchestra.run.smithi060.stdout:34359738376 2024-09-22T03:39:24.717 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:39:24.718 INFO:teuthology.orchestra.run.smithi060.stdout:42949672966 2024-09-22T03:39:24.718 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:39:24.723 INFO:teuthology.orchestra.run.smithi060.stdout:38654705671 2024-09-22T03:39:24.723 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:39:24.837 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.831+0000 7f17c5111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.840 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.839+0000 7f17c5111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.882 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.876+0000 7f895a6bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.878+0000 7f9111d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.884 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.883+0000 7f895a6bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:24.888 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:24.887+0000 7f9111d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:25.055 INFO:teuthology.orchestra.run.smithi060.stdout:34359738376 2024-09-22T03:39:25.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738376 for osd.1 2024-09-22T03:39:25.069 DEBUG:teuthology.parallel:result is None 2024-09-22T03:39:25.087 INFO:teuthology.orchestra.run.smithi060.stdout:38654705668 2024-09-22T03:39:25.101 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705668 for osd.0 2024-09-22T03:39:25.104 INFO:teuthology.orchestra.run.smithi060.stdout:42949672964 2024-09-22T03:39:25.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:39:26.102 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:39:26.118 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:39:26.231 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.223+0000 7ff8c1488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.233 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.232+0000 7ff8c1488640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.269 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.263+0000 7f2ba7f48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.271 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.270+0000 7f2ba7f48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.436 INFO:teuthology.orchestra.run.smithi060.stdout:38654705671 2024-09-22T03:39:26.449 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705671 got 38654705671 for osd.0 2024-09-22T03:39:26.449 DEBUG:teuthology.parallel:result is None 2024-09-22T03:39:26.469 INFO:teuthology.orchestra.run.smithi060.stdout:42949672966 2024-09-22T03:39:26.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672966 for osd.2 2024-09-22T03:39:26.482 DEBUG:teuthology.parallel:result is None 2024-09-22T03:39:26.482 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:39:26.482 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:39:26.636 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.631+0000 7f17e56fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.637 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.636+0000 7f17e56fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.834 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:26.834 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T03:39:26.846 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-22T03:39:25.992451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3456,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889728,"statfs":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590379,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":80,"num_write_kb":1107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.883092"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129496+0000","last_change":"2024-09-22T03:39:23.134202+0000","last_active":"2024-09-22T03:39:23.129496+0000","last_peered":"2024-09-22T03:39:23.129496+0000","last_clean":"2024-09-22T03:39:23.129496+0000","last_became_active":"2024-09-22T03:39:21.528163+0000","last_became_peered":"2024-09-22T03:39:21.528163+0000","last_unstale":"2024-09-22T03:39:23.129496+0000","last_undegraded":"2024-09-22T03:39:23.129496+0000","last_fullsized":"2024-09-22T03:39:23.129496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.131074+0000","last_change":"2024-09-22T03:39:23.137047+0000","last_active":"2024-09-22T03:39:23.131074+0000","last_peered":"2024-09-22T03:39:23.131074+0000","last_clean":"2024-09-22T03:39:23.131074+0000","last_became_active":"2024-09-22T03:39:21.643436+0000","last_became_peered":"2024-09-22T03:39:21.643436+0000","last_unstale":"2024-09-22T03:39:23.131074+0000","last_undegraded":"2024-09-22T03:39:23.131074+0000","last_fullsized":"2024-09-22T03:39:23.131074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.126438+0000","last_change":"2024-09-22T03:39:23.129284+0000","last_active":"2024-09-22T03:39:23.126438+0000","last_peered":"2024-09-22T03:39:23.126438+0000","last_clean":"2024-09-22T03:39:23.126438+0000","last_became_active":"2024-09-22T03:39:21.602834+0000","last_became_peered":"2024-09-22T03:39:21.602834+0000","last_unstale":"2024-09-22T03:39:23.126438+0000","last_undegraded":"2024-09-22T03:39:23.126438+0000","last_fullsized":"2024-09-22T03:39:23.126438+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129361+0000","last_change":"2024-09-22T03:39:23.134309+0000","last_active":"2024-09-22T03:39:23.129361+0000","last_peered":"2024-09-22T03:39:23.129361+0000","last_clean":"2024-09-22T03:39:23.129361+0000","last_became_active":"2024-09-22T03:39:21.581096+0000","last_became_peered":"2024-09-22T03:39:21.581096+0000","last_unstale":"2024-09-22T03:39:23.129361+0000","last_undegraded":"2024-09-22T03:39:23.129361+0000","last_fullsized":"2024-09-22T03:39:23.129361+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.161661+0000","last_change":"2024-09-22T03:39:23.161661+0000","last_active":"2024-09-22T03:39:23.161661+0000","last_peered":"2024-09-22T03:39:23.161661+0000","last_clean":"2024-09-22T03:39:23.161661+0000","last_became_active":"2024-09-22T03:39:21.505649+0000","last_became_peered":"2024-09-22T03:39:21.505649+0000","last_unstale":"2024-09-22T03:39:23.161661+0000","last_undegraded":"2024-09-22T03:39:23.161661+0000","last_fullsized":"2024-09-22T03:39:23.161661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.140794+0000","last_change":"2024-09-22T03:39:23.143430+0000","last_active":"2024-09-22T03:39:23.140794+0000","last_peered":"2024-09-22T03:39:23.140794+0000","last_clean":"2024-09-22T03:39:23.140794+0000","last_became_active":"2024-09-22T03:39:21.232739+0000","last_became_peered":"2024-09-22T03:39:21.232739+0000","last_unstale":"2024-09-22T03:39:23.140794+0000","last_undegraded":"2024-09-22T03:39:23.140794+0000","last_fullsized":"2024-09-22T03:39:23.140794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.140794+0000","last_change":"2024-09-22T03:39:23.143457+0000","last_active":"2024-09-22T03:39:23.140794+0000","last_peered":"2024-09-22T03:39:23.140794+0000","last_clean":"2024-09-22T03:39:23.140794+0000","last_became_active":"2024-09-22T03:39:21.238240+0000","last_became_peered":"2024-09-22T03:39:21.238240+0000","last_unstale":"2024-09-22T03:39:23.140794+0000","last_undegraded":"2024-09-22T03:39:23.140794+0000","last_fullsized":"2024-09-22T03:39:23.140794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129314+0000","last_change":"2024-09-22T03:39:23.136921+0000","last_active":"2024-09-22T03:39:23.129314+0000","last_peered":"2024-09-22T03:39:23.129314+0000","last_clean":"2024-09-22T03:39:23.129314+0000","last_became_active":"2024-09-22T03:39:21.293236+0000","last_became_peered":"2024-09-22T03:39:21.293236+0000","last_unstale":"2024-09-22T03:39:23.129314+0000","last_undegraded":"2024-09-22T03:39:23.129314+0000","last_fullsized":"2024-09-22T03:39:23.129314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.126140+0000","last_change":"2024-09-22T03:39:19.201092+0000","last_active":"2024-09-22T03:39:23.126140+0000","last_peered":"2024-09-22T03:39:23.126140+0000","last_clean":"2024-09-22T03:39:23.126140+0000","last_became_active":"2024-09-22T03:39:19.185454+0000","last_became_peered":"2024-09-22T03:39:19.185454+0000","last_unstale":"2024-09-22T03:39:23.126140+0000","last_undegraded":"2024-09-22T03:39:23.126140+0000","last_fullsized":"2024-09-22T03:39:23.126140+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:18.057924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:18.057924+0000","last_clean_scrub_stamp":"2024-09-22T03:39:18.057924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":464,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297264,"statfs":{"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1624,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296104,"statfs":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1368,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296360,"statfs":{"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:39:26.847 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:39:26.960 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.955+0000 7f1a2f991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:26.962 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:26.961+0000 7f1a2f991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:27.160 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:27.160 INFO:teuthology.orchestra.run.smithi060.stderr:dumped all 2024-09-22T03:39:27.173 INFO:teuthology.orchestra.run.smithi060.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-22T03:39:25.992451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3456,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889728,"statfs":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590379,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":80,"num_write_kb":1107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.883092"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129496+0000","last_change":"2024-09-22T03:39:23.134202+0000","last_active":"2024-09-22T03:39:23.129496+0000","last_peered":"2024-09-22T03:39:23.129496+0000","last_clean":"2024-09-22T03:39:23.129496+0000","last_became_active":"2024-09-22T03:39:21.528163+0000","last_became_peered":"2024-09-22T03:39:21.528163+0000","last_unstale":"2024-09-22T03:39:23.129496+0000","last_undegraded":"2024-09-22T03:39:23.129496+0000","last_fullsized":"2024-09-22T03:39:23.129496+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.131074+0000","last_change":"2024-09-22T03:39:23.137047+0000","last_active":"2024-09-22T03:39:23.131074+0000","last_peered":"2024-09-22T03:39:23.131074+0000","last_clean":"2024-09-22T03:39:23.131074+0000","last_became_active":"2024-09-22T03:39:21.643436+0000","last_became_peered":"2024-09-22T03:39:21.643436+0000","last_unstale":"2024-09-22T03:39:23.131074+0000","last_undegraded":"2024-09-22T03:39:23.131074+0000","last_fullsized":"2024-09-22T03:39:23.131074+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.126438+0000","last_change":"2024-09-22T03:39:23.129284+0000","last_active":"2024-09-22T03:39:23.126438+0000","last_peered":"2024-09-22T03:39:23.126438+0000","last_clean":"2024-09-22T03:39:23.126438+0000","last_became_active":"2024-09-22T03:39:21.602834+0000","last_became_peered":"2024-09-22T03:39:21.602834+0000","last_unstale":"2024-09-22T03:39:23.126438+0000","last_undegraded":"2024-09-22T03:39:23.126438+0000","last_fullsized":"2024-09-22T03:39:23.126438+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129361+0000","last_change":"2024-09-22T03:39:23.134309+0000","last_active":"2024-09-22T03:39:23.129361+0000","last_peered":"2024-09-22T03:39:23.129361+0000","last_clean":"2024-09-22T03:39:23.129361+0000","last_became_active":"2024-09-22T03:39:21.581096+0000","last_became_peered":"2024-09-22T03:39:21.581096+0000","last_unstale":"2024-09-22T03:39:23.129361+0000","last_undegraded":"2024-09-22T03:39:23.129361+0000","last_fullsized":"2024-09-22T03:39:23.129361+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.161661+0000","last_change":"2024-09-22T03:39:23.161661+0000","last_active":"2024-09-22T03:39:23.161661+0000","last_peered":"2024-09-22T03:39:23.161661+0000","last_clean":"2024-09-22T03:39:23.161661+0000","last_became_active":"2024-09-22T03:39:21.505649+0000","last_became_peered":"2024-09-22T03:39:21.505649+0000","last_unstale":"2024-09-22T03:39:23.161661+0000","last_undegraded":"2024-09-22T03:39:23.161661+0000","last_fullsized":"2024-09-22T03:39:23.161661+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.140794+0000","last_change":"2024-09-22T03:39:23.143430+0000","last_active":"2024-09-22T03:39:23.140794+0000","last_peered":"2024-09-22T03:39:23.140794+0000","last_clean":"2024-09-22T03:39:23.140794+0000","last_became_active":"2024-09-22T03:39:21.232739+0000","last_became_peered":"2024-09-22T03:39:21.232739+0000","last_unstale":"2024-09-22T03:39:23.140794+0000","last_undegraded":"2024-09-22T03:39:23.140794+0000","last_fullsized":"2024-09-22T03:39:23.140794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.140794+0000","last_change":"2024-09-22T03:39:23.143457+0000","last_active":"2024-09-22T03:39:23.140794+0000","last_peered":"2024-09-22T03:39:23.140794+0000","last_clean":"2024-09-22T03:39:23.140794+0000","last_became_active":"2024-09-22T03:39:21.238240+0000","last_became_peered":"2024-09-22T03:39:21.238240+0000","last_unstale":"2024-09-22T03:39:23.140794+0000","last_undegraded":"2024-09-22T03:39:23.140794+0000","last_fullsized":"2024-09-22T03:39:23.140794+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.129314+0000","last_change":"2024-09-22T03:39:23.136921+0000","last_active":"2024-09-22T03:39:23.129314+0000","last_peered":"2024-09-22T03:39:23.129314+0000","last_clean":"2024-09-22T03:39:23.129314+0000","last_became_active":"2024-09-22T03:39:21.293236+0000","last_became_peered":"2024-09-22T03:39:21.293236+0000","last_unstale":"2024-09-22T03:39:23.129314+0000","last_undegraded":"2024-09-22T03:39:23.129314+0000","last_fullsized":"2024-09-22T03:39:23.129314+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:20.064315+0000","last_clean_scrub_stamp":"2024-09-22T03:39:20.064315+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"13'52","reported_seq":87,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:39:23.126140+0000","last_change":"2024-09-22T03:39:19.201092+0000","last_active":"2024-09-22T03:39:23.126140+0000","last_peered":"2024-09-22T03:39:23.126140+0000","last_clean":"2024-09-22T03:39:23.126140+0000","last_became_active":"2024-09-22T03:39:19.185454+0000","last_became_peered":"2024-09-22T03:39:19.185454+0000","last_unstale":"2024-09-22T03:39:23.126140+0000","last_undegraded":"2024-09-22T03:39:23.126140+0000","last_fullsized":"2024-09-22T03:39:23.126140+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:39:18.057924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:39:18.057924+0000","last_clean_scrub_stamp":"2024-09-22T03:39:18.057924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672966,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":464,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297264,"statfs":{"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1624,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296104,"statfs":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1368,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296360,"statfs":{"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:39:27.173 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:39:27.174 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:39:27.174 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:39:27.174 DEBUG:teuthology.orchestra.run.smithi060:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:39:27.287 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:27.283+0000 7f670eea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:27.288 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:39:27.287+0000 7f670eea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:39:27.526 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:27.527 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:39:27.540 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:39:27.540 INFO:teuthology.run_tasks:Running task cbt... 2024-09-22T03:39:27.553 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi060.front.sepia.ceph.com', 'osds': ['ubuntu@smithi060.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi060.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi060.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi060.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-22T03:39:27.553 DEBUG:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-22T03:39:27.571 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:39:27.571 DEBUG:teuthology.orchestra.run.smithi060:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:39:27.625 DEBUG:teuthology.orchestra.run.smithi060:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-22T03:39:27.681 INFO:teuthology.orchestra.run.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-22T03:39:27.989 INFO:tasks.cbt:Installing collectl 2024-09-22T03:39:27.989 DEBUG:teuthology.orchestra.run.smithi060:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-22T03:39:28.081 INFO:teuthology.orchestra.run.smithi060.stderr:--2024-09-22 03:39:28-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-22T03:39:28.256 INFO:teuthology.orchestra.run.smithi060.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-09-22T03:39:28.269 INFO:teuthology.orchestra.run.smithi060.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-09-22T03:39:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:39:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75FwxY9PDWJ-KK81njDDvEe_Tr_436st-q9LFoKkIr_Idhl2kmQ0Y-Pl8WDP3eDbWLSfEyQg2x_NRjCKQIP6HXYrqQ%3D%3D&use_mirror=pilotfiber&r= [following] 2024-09-22T03:39:28.502 INFO:teuthology.orchestra.run.smithi060.stderr:--2024-09-22 03:39:28-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75FwxY9PDWJ-KK81njDDvEe_Tr_436st-q9LFoKkIr_Idhl2kmQ0Y-Pl8WDP3eDbWLSfEyQg2x_NRjCKQIP6HXYrqQ%3D%3D&use_mirror=pilotfiber&r= 2024-09-22T03:39:28.513 INFO:teuthology.orchestra.run.smithi060.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-22T03:39:28.586 INFO:teuthology.orchestra.run.smithi060.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-22T03:39:28.900 INFO:teuthology.orchestra.run.smithi060.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:39:28.900 INFO:teuthology.orchestra.run.smithi060.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-22T03:39:28.900 INFO:teuthology.orchestra.run.smithi060.stderr:--2024-09-22 03:39:28-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-22T03:39:28.913 INFO:teuthology.orchestra.run.smithi060.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2024-09-22T03:39:28.927 INFO:teuthology.orchestra.run.smithi060.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2024-09-22T03:39:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-22T03:39:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-22T03:39:29.027 INFO:teuthology.orchestra.run.smithi060.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-22T03:39:29.027 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:29.052 INFO:teuthology.orchestra.run.smithi060.stderr: 0K .......... .......... .......... .......... .......... 8% 1.94M 0s 2024-09-22T03:39:29.062 INFO:teuthology.orchestra.run.smithi060.stderr: 50K .......... .......... .......... .......... .......... 16% 4.83M 0s 2024-09-22T03:39:29.065 INFO:teuthology.orchestra.run.smithi060.stderr: 100K .......... .......... .......... .......... .......... 24% 18.1M 0s 2024-09-22T03:39:29.067 INFO:teuthology.orchestra.run.smithi060.stderr: 150K .......... .......... .......... .......... .......... 32% 20.5M 0s 2024-09-22T03:39:29.075 INFO:teuthology.orchestra.run.smithi060.stderr: 200K .......... .......... .......... .......... .......... 41% 6.01M 0s 2024-09-22T03:39:29.078 INFO:teuthology.orchestra.run.smithi060.stderr: 250K .......... .......... .......... .......... .......... 49% 22.8M 0s 2024-09-22T03:39:29.078 INFO:teuthology.orchestra.run.smithi060.stderr: 300K .......... .......... .......... .......... .......... 57% 296M 0s 2024-09-22T03:39:29.080 INFO:teuthology.orchestra.run.smithi060.stderr: 350K .......... .......... .......... .......... .......... 65% 23.7M 0s 2024-09-22T03:39:29.089 INFO:teuthology.orchestra.run.smithi060.stderr: 400K .......... .......... .......... .......... .......... 74% 6.02M 0s 2024-09-22T03:39:29.089 INFO:teuthology.orchestra.run.smithi060.stderr: 450K .......... .......... .......... .......... .......... 82% 172M 0s 2024-09-22T03:39:29.089 INFO:teuthology.orchestra.run.smithi060.stderr: 500K .......... .......... .......... .......... .......... 90% 297M 0s 2024-09-22T03:39:29.090 INFO:teuthology.orchestra.run.smithi060.stderr: 550K .......... .......... .......... .......... .......... 98% 26.2M 0s 2024-09-22T03:39:29.090 INFO:teuthology.orchestra.run.smithi060.stderr: 600K ...... 100% 11.4T=0.06s 2024-09-22T03:39:29.091 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:29.091 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22 03:39:29 (9.35 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-22T03:39:29.091 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:39:29.138 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/ 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/collectl.conf 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/ 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Socket.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/colmux.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Import.html 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/style.css 2024-09-22T03:39:29.139 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Process.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-22T03:39:29.140 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Export.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Startup.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Performance.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Messages.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Logging.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-22T03:39:29.141 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Misc.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Examples.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Data.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-22T03:39:29.142 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Playback.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Hello.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Network.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Memory.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/Disks.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/service/ 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/service/collectl.service 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/initd/ 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-22T03:39:29.143 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/initd/collectl 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/misc.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/gexpr.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/statsd.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/envrules.std 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/hello.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/vmsum.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/proctree.ph 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/README-WINDOWS 2024-09-22T03:39:29.144 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/formatit.ph 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/colmux 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/INSTALL 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/graphite.ph 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/vmstat.ph 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/README 2024-09-22T03:39:29.146 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/man1/ 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/man1/collectl.1 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/man1/colmux.1 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/lexpr.ph 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/client.pl 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/collectl 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/COPYING 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/vnet.ph 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/GPL 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/UNINSTALL 2024-09-22T03:39:29.147 INFO:teuthology.orchestra.run.smithi060.stdout:collectl-4.3.1/ARTISTIC 2024-09-22T03:39:29.204 INFO:teuthology.orchestra.run.smithi060.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-22T03:39:29.206 DEBUG:teuthology.orchestra.run.smithi060:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-22T03:39:29.829 INFO:teuthology.orchestra.run.smithi060.stdout:Last metadata expiration check: 0:03:05 ago on Sun 22 Sep 2024 03:36:24 AM UTC. 2024-09-22T03:39:29.924 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-22T03:39:29.927 INFO:teuthology.orchestra.run.smithi060.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-22T03:39:29.978 INFO:teuthology.orchestra.run.smithi060.stdout:Dependencies resolved. 2024-09-22T03:39:29.979 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:39:29.979 INFO:teuthology.orchestra.run.smithi060.stdout: Package Arch Version Repository Size 2024-09-22T03:39:29.979 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:39:29.979 INFO:teuthology.orchestra.run.smithi060.stdout:Installing: 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout:Installing dependencies: 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction Summary 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout:================================================================================ 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout:Install 5 Packages 2024-09-22T03:39:29.980 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:29.981 INFO:teuthology.orchestra.run.smithi060.stdout:Total download size: 2.4 M 2024-09-22T03:39:29.981 INFO:teuthology.orchestra.run.smithi060.stdout:Installed size: 11 M 2024-09-22T03:39:29.981 INFO:teuthology.orchestra.run.smithi060.stdout:Downloading Packages: 2024-09-22T03:39:30.450 INFO:teuthology.orchestra.run.smithi060.stdout:(1/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 157 kB/s | 24 kB 00:00 2024-09-22T03:39:30.473 INFO:teuthology.orchestra.run.smithi060.stdout:(2/5): libradospp-devel-19.3.0-5092.g62da54b4.e 174 kB/s | 31 kB 00:00 2024-09-22T03:39:30.524 INFO:teuthology.orchestra.run.smithi060.stdout:(3/5): pdsh-2.34-7.el9.x86_64.rpm 1.5 MB/s | 118 kB 00:00 2024-09-22T03:39:30.549 INFO:teuthology.orchestra.run.smithi060.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 174 kB/s | 13 kB 00:00 2024-09-22T03:39:31.324 INFO:teuthology.orchestra.run.smithi060.stdout:(5/5): perf-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 2.2 MB 00:01 2024-09-22T03:39:31.327 INFO:teuthology.orchestra.run.smithi060.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:39:31.328 INFO:teuthology.orchestra.run.smithi060.stdout:Total 1.8 MB/s | 2.4 MB 00:01 2024-09-22T03:39:31.360 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction check 2024-09-22T03:39:31.386 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction check succeeded. 2024-09-22T03:39:31.386 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction test 2024-09-22T03:39:31.525 INFO:teuthology.orchestra.run.smithi060.stdout:Transaction test succeeded. 2024-09-22T03:39:31.525 INFO:teuthology.orchestra.run.smithi060.stdout:Running transaction 2024-09-22T03:39:31.867 INFO:teuthology.orchestra.run.smithi060.stdout: Preparing : 1/1 2024-09-22T03:39:31.927 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-22T03:39:31.957 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-22T03:39:31.988 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-22T03:39:32.339 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-22T03:39:32.421 INFO:teuthology.orchestra.run.smithi060.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:39:34.156 INFO:teuthology.orchestra.run.smithi060.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:39:34.156 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-22T03:39:34.157 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-22T03:39:34.157 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-22T03:39:34.157 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout:Installed: 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: pdsh-2.34-7.el9.x86_64 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: perf-5.14.0-508.el9.x86_64 2024-09-22T03:39:34.500 INFO:teuthology.orchestra.run.smithi060.stdout: 2024-09-22T03:39:34.501 INFO:teuthology.orchestra.run.smithi060.stdout:Complete! 2024-09-22T03:39:34.836 INFO:tasks.cbt:benchmark: radosbench 2024-09-22T03:39:34.836 DEBUG:teuthology.orchestra.run.smithi060:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-22T03:39:34.862 DEBUG:teuthology.orchestra.run.smithi060:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - Settings.cluster: 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 'clients': ['ubuntu@smithi060.front.sepia.ceph.com'], 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 'head': 'ubuntu@smithi060.front.sepia.ceph.com', 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 'is_teuthology': True, 2024-09-22T03:39:35.276 INFO:teuthology.orchestra.run.smithi060.stderr: 'iterations': 1, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'mons': ['ubuntu@smithi060.front.sepia.ceph.com'], 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'osds': ['ubuntu@smithi060.front.sepia.ceph.com'], 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'osds_per_node': 3, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'pid_dir': '/var/run/ceph', 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'pgp_size': 256, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'replication': 'replicated'}}, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'rebuild_every_test': False, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'rgws': {'ubuntu@smithi060.front.sepia.ceph.com': None}, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'use_existing': True, 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr: 'user': 'ubuntu'} 2024-09-22T03:39:35.277 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2024-09-22T03:39:35.297 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-22T03:39:35.298 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-09-22T03:39:35.302 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:39:35.303 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-09-22T03:39:35.306 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:39:35.306 INFO:teuthology.orchestra.run.smithi060.stderr:03:39:35 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:40:35.356 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:40:35.401 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:40:35.443 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-09-22T03:40:35.446 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-09-22T03:40:35.447 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-09-22T03:40:35.463 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi060.front.sepia.ceph.com ; done 2024-09-22T03:40:35.468 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-09-22T03:40:35.491 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - WARNING - cbt - sudo killall -9 rados 2024-09-22T03:40:35.491 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - WARNING - cbt - error 1 seen, continuing anyway... 2024-09-22T03:40:35.491 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2024-09-22T03:40:35.511 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2024-09-22T03:40:35.512 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-22T03:40:35.512 INFO:teuthology.orchestra.run.smithi060.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3 2024-09-22T03:40:35.512 INFO:teuthology.orchestra.run.smithi060.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': True, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2024-09-22T03:40:35.512 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-22T03:40:35.512 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-22T03:40:35.526 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-09-22T03:40:35.529 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:40:35.530 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-09-22T03:40:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:40:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:40:35.534 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-09-22T03:40:35.853 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-09-22T03:40:36.181 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-09-22T03:40:37.184 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:37 - INFO - cbt - Checking Health after pool creation. 2024-09-22T03:40:42.190 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:40:42.669 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:42 - INFO - cbt - Final Pool Health Check. 2024-09-22T03:40:47.674 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:40:48.042 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:40:48.068 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:40:48.101 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-09-22T03:40:48.103 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-09-22T03:40:48.104 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-22T03:40:48.476 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-22T03:40:49.040 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-22T03:40:49.412 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-09-22T03:40:49.469 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-09-22T03:40:50.323 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-09-22T03:40:50.326 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:40:50.327 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-09-22T03:40:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:40:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:40:50.330 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - INFO - cbt - Running radosbench write test. 2024-09-22T03:40:50.331 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-09-22T03:40:50.332 INFO:teuthology.orchestra.run.smithi060.stderr:03:40:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-09-22T03:41:57.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:42:03.089 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-22T03:42:08.493 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2024-09-22T03:42:13.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2024-09-22T03:42:19.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2024-09-22T03:42:24.507 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:42:24.535 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:42:24.570 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-09-22T03:42:24.572 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-09-22T03:42:24.574 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-09-22T03:42:24.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2024-09-22T03:42:24.887 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-09-22T03:42:24.901 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi060.front.sepia.ceph.com ; done 2024-09-22T03:42:24.931 INFO:teuthology.orchestra.run.smithi060.stderr:03:42:24 - INFO - cbt - Convert results to json format. 2024-09-22T03:42:24.946 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-09-22T03:42:24.955 DEBUG:teuthology.orchestra.run.smithi060:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-09-22T03:42:25.110 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-09-22T03:42:25.110 INFO:tasks.cbt_performance:reading cbt preferences 2024-09-22T03:42:25.110 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-09-22T03:42:25.157 INFO:teuthology.orchestra.run.smithi060.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi060.front.sepia.ceph.com 2024-09-22T03:42:25.157 INFO:teuthology.orchestra.run.smithi060.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi060.front.sepia.ceph.com 2024-09-22T03:42:25.158 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:42:25.158 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi060.front.sepia.ceph.com of=/dev/stdout 2024-09-22T03:42:25.175 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T03:42:25.175 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-09-22T03:42:25.231 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T03:42:25.231 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T03:42:25.231 DEBUG:teuthology.orchestra.run.smithi060:> set -ex 2024-09-22T03:42:25.231 DEBUG:teuthology.orchestra.run.smithi060:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi060.front.sepia.ceph.com of=/dev/stdout 2024-09-22T03:42:25.287 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T03:42:25.287 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-09-22T03:42:25.343 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T03:42:25.343 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T03:42:25.351 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T03:42:25.358 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T03:42:25.358 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:42:25.367 DEBUG:teuthology.orchestra.run.smithi060:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:42:25.514 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:42:25.506+0000 7efd61791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:42:25.515 INFO:teuthology.orchestra.run.smithi060.stderr:2024-09-22T03:42:25.514+0000 7efd61791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:42:25.744 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:42:25.744 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:42:25.744 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:42:25.744 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:42:25.845 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:42:25.845 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:42:25.845 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:42:25.943 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:42:25.943 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:42:25.943 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:42:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:42:25.944 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-09-22T03:42:25.944 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:42:25.945 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:42:25.945 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:42:25.945 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:42:25.975 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:42:25.975 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:42:25.975 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:42:25.975 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:42:26.026 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:42:26.027 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:42:26.027 DEBUG:teuthology.orchestra.run.smithi060:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-22T03:42:26.107 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:42:26.107 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:42:26.381 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:42:26.381 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:42:26.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:42:26.473 DEBUG:teuthology.orchestra.run.smithi060:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:42:26.532 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:42:26.533 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359/data/mon.a.tgz 2024-09-22T03:42:26.533 DEBUG:teuthology.orchestra.run.smithi060:> mktemp 2024-09-22T03:42:26.556 INFO:teuthology.orchestra.run.smithi060.stdout:/tmp/tmp.3CMP7p9B0g 2024-09-22T03:42:26.556 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.3CMP7p9B0g 2024-09-22T03:42:26.691 DEBUG:teuthology.orchestra.run.smithi060:> sudo chmod 0666 /tmp/tmp.3CMP7p9B0g 2024-09-22T03:42:26.799 DEBUG:teuthology.orchestra.remote:smithi060:/tmp/tmp.3CMP7p9B0g is 210KB 2024-09-22T03:42:26.893 DEBUG:teuthology.orchestra.run.smithi060:> rm -fr /tmp/tmp.3CMP7p9B0g 2024-09-22T03:42:26.908 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:42:26.908 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:42:27.035 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:42:27.036 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:42:27.037 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359/remote/smithi060/crash 2024-09-22T03:42:27.037 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:42:27.101 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:42:27.101 DEBUG:teuthology.orchestra.run.smithi060:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:42:27.176 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:42:27.177 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29623.log 2024-09-22T03:42:27.177 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29119.log 2024-09-22T03:42:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29559.log 2024-09-22T03:42:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29413.log/var/log/ceph/ceph-client.admin.29119.log: 2024-09-22T03:42:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:42:27.178 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2024-09-22T03:42:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28343.log 2024-09-22T03:42:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27389.log 2024-09-22T03:42:27.179 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30573.log 2024-09-22T03:42:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28431.log 2024-09-22T03:42:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29327.log 2024-09-22T03:42:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29539.log 2024-09-22T03:42:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:42:27.183 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28968.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29707.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28581.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29024.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31118.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29245.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29203.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29326.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29452.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29068.log 2024-09-22T03:42:27.184 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28541.log 2024-09-22T03:42:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:42:27.185 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29161.log 2024-09-22T03:42:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:42:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29325.log 2024-09-22T03:42:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29665.log 2024-09-22T03:42:27.186 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30637.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30774.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30687.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30710.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30664.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28790.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30775.log 2024-09-22T03:42:27.187 INFO:teuthology.orchestra.run.smithi060.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28627.log 2024-09-22T03:42:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29413.log: /var/log/ceph/ceph-client.admin.29559.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.28790.log: /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.28627.log: /var/log/ceph/ceph-client.admin.31118.log: /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.30774.log: /var/log/ceph/ceph-client.admin.30775.log: /var/log/ceph/ceph-client.admin.28343.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31118.log.gz 2024-09-22T03:42:27.195 INFO:teuthology.orchestra.run.smithi060.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30774.log.gz 2024-09-22T03:42:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.28820.log: /var/log/ceph/ceph-client.admin.29327.log: /var/log/ceph/ceph.tmp-client.admin.27389.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30775.log.gz 2024-09-22T03:42:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29325.log: /var/log/ceph/ceph-client.admin.30637.log: /var/log/ceph/ceph-client.admin.28431.log: /var/log/ceph/ceph-client.admin.29024.log: /var/log/ceph/ceph-client.admin.29707.log: /var/log/ceph/ceph-client.admin.29539.log: /var/log/ceph/ceph-client.admin.28968.log: /var/log/ceph/ceph-client.admin.30687.log: /var/log/ceph/ceph-client.admin.30664.log: 84.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:42:27.195 INFO:teuthology.orchestra.run.smithi060.stderr:/var/log/ceph/ceph-client.admin.29452.log: /var/log/ceph/ceph-client.admin.28581.log: /var/log/ceph/ceph-client.admin.29453.log: /var/log/ceph/ceph-client.admin.29245.log: /var/log/ceph/ceph-client.admin.30573.log: /var/log/ceph/ceph-client.admin.29203.log: /var/log/ceph/ceph-client.admin.29326.log: /var/log/ceph/ceph-client.admin.29068.log: /var/log/ceph/ceph-client.admin.28541.log: /var/log/ceph/ceph-client.admin.29161.log: /var/log/ceph/ceph-client.admin.30710.log: /var/log/ceph/ceph-client.admin.29665.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28343.log.gz 2024-09-22T03:42:27.202 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30664.log.gz 2024-09-22T03:42:27.202 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30687.log.gz 2024-09-22T03:42:27.202 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30710.log.gz 2024-09-22T03:42:27.204 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29119.log.gz 2024-09-22T03:42:27.206 INFO:teuthology.orchestra.run.smithi060.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:42:27.208 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28541.log.gz 2024-09-22T03:42:27.209 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28581.log.gz 2024-09-22T03:42:27.209 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28627.log.gz 2024-09-22T03:42:27.211 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28790.log.gz 2024-09-22T03:42:27.211 INFO:teuthology.orchestra.run.smithi060.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2024-09-22T03:42:27.213 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28968.log.gz 2024-09-22T03:42:27.213 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29024.log.gz 2024-09-22T03:42:27.213 INFO:teuthology.orchestra.run.smithi060.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.29068.log.gz 2024-09-22T03:42:27.228 INFO:teuthology.orchestra.run.smithi060.stderr: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27389.log.gz 2024-09-22T03:42:27.256 INFO:teuthology.orchestra.run.smithi060.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:42:27.259 INFO:teuthology.orchestra.run.smithi060.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:42:27.262 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28431.log.gz 2024-09-22T03:42:27.379 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30573.log.gz 2024-09-22T03:42:27.379 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30637.log.gz 2024-09-22T03:42:27.413 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29161.log.gz 2024-09-22T03:42:27.413 INFO:teuthology.orchestra.run.smithi060.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29203.log.gz 2024-09-22T03:42:27.413 INFO:teuthology.orchestra.run.smithi060.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29245.log.gz 2024-09-22T03:42:27.413 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29327.log.gz 2024-09-22T03:42:27.413 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29325.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29326.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29413.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29452.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29539.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29559.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29623.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29665.log.gz 2024-09-22T03:42:27.414 INFO:teuthology.orchestra.run.smithi060.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29707.log.gz 2024-09-22T03:42:29.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T03:42:30.117 INFO:teuthology.orchestra.run.smithi060.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:42:30.535 INFO:teuthology.orchestra.run.smithi060.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T03:42:31.437 INFO:teuthology.orchestra.run.smithi060.stderr: 95.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:42:31.439 INFO:teuthology.orchestra.run.smithi060.stderr: 2024-09-22T03:42:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:real 0m4.283s 2024-09-22T03:42:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:user 0m4.091s 2024-09-22T03:42:31.439 INFO:teuthology.orchestra.run.smithi060.stderr:sys 0m0.186s 2024-09-22T03:42:31.439 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:42:31.440 DEBUG:teuthology.misc:Transferring archived files from smithi060:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359/remote/smithi060/log 2024-09-22T03:42:31.441 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:42:32.253 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:42:32.261 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:42:32.262 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:42:32.315 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:42:32.328 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:42:32.328 DEBUG:teuthology.orchestra.run.smithi060:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:42:32.345 INFO:teuthology.orchestra.run.smithi060.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:42:32.404 INFO:teuthology.orchestra.run.smithi060.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:42:32.404 INFO:teuthology.orchestra.run.smithi060.stdout:=============================================================================== 2024-09-22T03:42:32.404 INFO:teuthology.orchestra.run.smithi060.stdout:^* hv01.front.sepia.ceph.com 3 6 377 57 -289us[ -291us] +/- 59ms 2024-09-22T03:42:32.405 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 58 +617us[ +615us] +/- 53ms 2024-09-22T03:42:32.405 INFO:teuthology.orchestra.run.smithi060.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 -402us[ -404us] +/- 68ms 2024-09-22T03:42:32.405 INFO:teuthology.orchestra.run.smithi060.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:42:32.425 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:42:32.434 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:42:32.434 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:42:32.484 DEBUG:teuthology.orchestra.run.smithi060:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:42:32.639 DEBUG:teuthology.orchestra.run.smithi060:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T03:42:32.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:42:32.668 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:42:32.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:42:32.718 INFO:teuthology.task.internal:Duration was 615.092607 seconds 2024-09-22T03:42:32.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:42:32.727 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:42:32.727 DEBUG:teuthology.orchestra.run.smithi060:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:42:32.848 INFO:teuthology.orchestra.run.smithi060.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:42:33.158 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:42:33.159 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi060.front.sepia.ceph.com 2024-09-22T03:42:33.159 DEBUG:teuthology.orchestra.run.smithi060:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:42:33.180 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:42:33.180 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:42:33.249 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:42:33.249 DEBUG:teuthology.orchestra.run.smithi060:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:42:33.400 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:42:33.414 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:42:33.415 DEBUG:teuthology.orchestra.run.smithi060:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:42:33.445 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:42:33.455 DEBUG:teuthology.orchestra.run.smithi060:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:42:33.516 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = core 2024-09-22T03:42:33.534 DEBUG:teuthology.orchestra.run.smithi060:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:42:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:42:33.590 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:42:33.599 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:42:33.600 DEBUG:teuthology.misc:Transferring archived files from smithi060:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914359/remote/smithi060 2024-09-22T03:42:33.600 DEBUG:teuthology.orchestra.run.smithi060:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:42:33.758 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:42:33.758 DEBUG:teuthology.orchestra.run.smithi060:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:42:33.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:42:33.786 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:42:33.786 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:42:33.795 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:42:33.795 DEBUG:teuthology.orchestra.run.smithi060:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:42:33.830 INFO:teuthology.orchestra.run.smithi060.stdout: 265899 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:42 /home/ubuntu/cephtest 2024-09-22T03:42:33.831 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:42:33.841 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:42:33.862 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7914359 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_write} duration: 615.0926065444946 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T03:42:33.862 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:42:33.941 INFO:teuthology.run:pass