2024-09-21T21:19:34.198 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T21:19:34.218 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:19:34.304 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4K_rand_read} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914348' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk8KP3XpemVFVE2VMkzXmkzVvwLe/vw8QtLyjjkejqHYqoT8AK605BMuMP2Cb5X0dffaypgvtlHQuKX3DISWlQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T21:19:34.304 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T21:19:34.305 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T21:19:34.305 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T21:19:34.306 INFO:teuthology.task.internal:Checking packages... 2024-09-21T21:19:34.328 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T21:19:34.328 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T21:19:34.328 INFO:teuthology.packaging:ref: None 2024-09-21T21:19:34.328 INFO:teuthology.packaging:tag: None 2024-09-21T21:19:34.328 INFO:teuthology.packaging:branch: main 2024-09-21T21:19:34.328 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:19:34.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T21:19:34.538 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T21:19:34.540 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T21:19:34.551 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T21:19:34.551 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T21:19:34.593 INFO:teuthology.task.internal:Saving configuration 2024-09-21T21:19:34.611 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T21:19:34.627 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T21:19:34.653 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:14:42.805890', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk8KP3XpemVFVE2VMkzXmkzVvwLe/vw8QtLyjjkejqHYqoT8AK605BMuMP2Cb5X0dffaypgvtlHQuKX3DISWlQ='} 2024-09-21T21:19:34.653 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T21:19:34.664 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-21T21:19:34.664 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T21:19:34.716 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2467354160>, signals=[15]) 2024-09-21T21:19:34.716 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T21:19:34.727 INFO:teuthology.task.internal:Opening connections... 2024-09-21T21:19:34.727 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:19:34.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:19:34.802 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T21:19:34.814 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-09-21T21:19:34.881 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-09-21T21:19:34.881 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-09-21T21:19:34.939 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-09-21T21:19:34.940 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T21:19:34.940 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-09-21T21:19:34.940 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T21:19:34.940 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T21:19:34.940 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T21:19:34.940 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-09-21T21:19:34.959 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T21:19:34.967 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T21:19:34.987 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T21:19:34.987 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:19:35.003 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T21:19:35.014 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T21:19:35.014 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:19:35.083 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T21:19:35.113 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T21:19:35.129 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T21:19:35.129 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T21:19:35.129 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:19:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:19:35.147 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-21T21:19:35.204 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-503.el9.x86_64 2024-09-21T21:19:35.205 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-503.el9.x86_64 2024-09-21T21:19:35.205 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-09-21T21:19:39.963 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.3 MB 00:03 2024-09-21T21:19:41.756 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 28 MB/s | 20 MB 00:00 2024-09-21T21:19:45.965 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-21T21:19:47.482 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-21T21:19:48.678 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-21T21:19:54.184 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T21:19:56.165 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T21:19:56.165 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T21:19:56.253 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:19:56.260 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:19:56.260 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-21T21:19:56.260 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-09-21T21:19:56.261 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:19:56.262 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 79 M 2024-09-21T21:19:56.262 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 126 M 2024-09-21T21:19:56.262 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:19:56.393 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 423 kB/s | 36 kB 00:00 2024-09-21T21:19:56.785 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-21T21:19:57.202 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 39 MB/s | 35 MB 00:00 2024-09-21T21:19:57.528 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-21T21:19:57.528 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:57.528 INFO:teuthology.orchestra.run.smithi181.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-21T21:19:58.212 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:19:58.566 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:19:58.566 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:19:59.564 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:19:59.565 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:20:02.013 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:20:02.967 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:20:03.069 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:20:05.837 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:20:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:20:10.795 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:20:10.840 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:15.674 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:53.738 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:53.778 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:54.156 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:54.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:20:54.156 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:20:54.157 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:20:55.060 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:20:55.444 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:20:55.528 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.529 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:20:55.529 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T21:20:55.529 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:20:55.529 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:20:55.529 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T21:20:55.529 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:20:55.529 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:20:55.588 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:20:55.588 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:20:55.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 82.el9 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T21:20:55.621 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8274670 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-09-21T21:20:55.622 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T21:20:55.623 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T21:20:55.624 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-09-21T21:20:55.624 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:20:55.624 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T21:20:55.625 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T21:20:55.625 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-508.el9.x86_64 2024-09-21T21:20:55.625 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T21:20:56.217 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-09-21T21:20:57.465 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T21:20:57.492 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-09-21T21:20:57.495 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-21T21:20:57.511 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.QNgqQmxw7T 2024-09-21T21:20:57.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QNgqQmxw7T 2024-09-21T21:20:57.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.QNgqQmxw7T 2024-09-21T21:20:57.712 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.QNgqQmxw7T is 6KB 2024-09-21T21:20:57.760 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.QNgqQmxw7T 2024-09-21T21:20:57.775 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T21:20:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T21:20:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T21:20:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T21:20:57.838 INFO:teuthology.orchestra.run.smithi181.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T21:20:57.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T21:20:58.016 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-09-21T21:21:28.046 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T21:21:28.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T21:21:28.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:22:28.104 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T21:22:37.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T21:22:37.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:22:40.447 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-21T21:22:40.858 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T21:22:40.858 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T21:22:40.858 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-09-21T21:22:40.875 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-508.el9.x86_64 2024-09-21T21:22:40.875 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T21:22:40.876 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:22:40.876 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T21:22:41.877 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T21:22:41.877 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:22:41.972 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-09-21T21:22:41.992 DEBUG:teuthology.parallel:result is None 2024-09-21T21:22:41.993 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T21:22:41.999 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T21:22:41.999 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:22:42.013 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T21:22:42.057 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T21:22:42.091 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T21:22:42.091 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:22:42.109 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T21:22:42.124 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T21:22:42.125 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:22:42.174 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:22:42.184 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:22:42.186 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T21:22:42.193 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T21:22:42.193 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:22:42.251 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T21:22:42.259 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T21:22:42.260 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:22:42.306 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:22:42.379 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:22:42.430 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:22:42.513 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:22:42.552 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:22:42.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:22:42.618 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-09-21T21:22:42.714 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:22:42.989 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T21:22:43.007 INFO:teuthology.task.internal:Starting timer... 2024-09-21T21:22:43.007 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T21:22:43.061 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T21:22:43.144 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-09-21T21:22:43.177 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-09-21T21:22:43.179 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T21:22:43.179 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-09-21T21:22:43.237 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-09-21T21:22:43.237 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-09-21T21:22:43.238 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:22:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:22:43.303 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T21:22:43.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-09-21T21:22:43.378 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T21:22:43.387 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T21:22:43.397 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T21:22:43.398 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T21:26:24.944 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-09-21T21:26:24.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T21:26:24.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:26:25.020 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-09-21T21:26:25.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-09-21T21:26:25.063 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T21:26:25.075 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T21:26:25.075 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T21:26:25.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:26:25.133 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T21:26:25.148 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T21:26:25.172 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-09-21T21:26:25.183 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-09-21T21:26:25.197 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T21:26:25.211 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T21:26:25.268 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:26:25.272 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:26:25.273 INFO:teuthology.run_tasks:Running task install... 2024-09-21T21:26:25.280 DEBUG:teuthology.task.install:project ceph 2024-09-21T21:26:25.280 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T21:26:25.280 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T21:26:25.281 INFO:teuthology.task.install:Using flavor: crimson 2024-09-21T21:26:25.285 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T21:26:25.285 INFO:teuthology.task.install:extra packages: [] 2024-09-21T21:26:25.286 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T21:26:25.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:26:25.431 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-21T21:26:25.432 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-21T21:26:25.603 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T21:26:25.603 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:26:25.603 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T21:26:25.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T21:26:25.630 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T21:26:25.697 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T21:26:25.773 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T21:26:25.838 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-09-21T21:26:25.840 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-09-21T21:26:26.078 INFO:teuthology.orchestra.run.smithi181.stdout:45 files removed 2024-09-21T21:26:26.105 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-09-21T21:26:26.670 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 317 kB/s | 88 kB 00:00 2024-09-21T21:26:26.970 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-21T21:26:27.246 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 8.7 kB/s | 2.1 kB 00:00 2024-09-21T21:26:27.746 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-21T21:26:29.505 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T21:26:33.748 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-21T21:26:35.607 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-21T21:26:37.103 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-21T21:26:42.509 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-21T21:26:44.761 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-09-21T21:26:44.780 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T21:26:44.781 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T21:26:44.782 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:Installing weak dependencies: 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:Install 31 Packages 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:26:44.783 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 60 M 2024-09-21T21:26:44.784 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:26:45.852 INFO:teuthology.orchestra.run.smithi181.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 7.5 MB/s | 6.1 MB 00:00 2024-09-21T21:26:45.902 INFO:teuthology.orchestra.run.smithi181.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 504 kB/s | 25 kB 00:00 2024-09-21T21:26:45.977 INFO:teuthology.orchestra.run.smithi181.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 11 MB/s | 808 kB 00:00 2024-09-21T21:26:46.111 INFO:teuthology.orchestra.run.smithi181.stdout:(4/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 9.7 MB/s | 10 MB 00:01 2024-09-21T21:26:46.153 INFO:teuthology.orchestra.run.smithi181.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 3.1 MB/s | 558 kB 00:00 2024-09-21T21:26:46.212 INFO:teuthology.orchestra.run.smithi181.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 773 kB/s | 45 kB 00:00 2024-09-21T21:26:46.279 INFO:teuthology.orchestra.run.smithi181.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 2.2 MB/s | 148 kB 00:00 2024-09-21T21:26:46.345 INFO:teuthology.orchestra.run.smithi181.stdout:(8/33): python3-cephfs-19.3.0-5092.g62da54b4.el 2.3 MB/s | 159 kB 00:00 2024-09-21T21:26:46.462 INFO:teuthology.orchestra.run.smithi181.stdout:(9/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_6 15 MB/s | 5.4 MB 00:00 2024-09-21T21:26:46.487 INFO:teuthology.orchestra.run.smithi181.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 2.2 MB/s | 314 kB 00:00 2024-09-21T21:26:46.521 INFO:teuthology.orchestra.run.smithi181.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-21T21:26:46.547 INFO:teuthology.orchestra.run.smithi181.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T21:26:46.747 INFO:teuthology.orchestra.run.smithi181.stdout:(13/33): ceph-common-19.3.0-5092.g62da54b4.el9. 12 MB/s | 20 MB 00:01 2024-09-21T21:26:47.347 INFO:teuthology.orchestra.run.smithi181.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 129 kB/s | 107 kB 00:00 2024-09-21T21:26:47.397 INFO:teuthology.orchestra.run.smithi181.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 201 kB/s | 171 kB 00:00 2024-09-21T21:26:47.456 INFO:teuthology.orchestra.run.smithi181.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 226 kB/s | 160 kB 00:00 2024-09-21T21:26:47.923 INFO:teuthology.orchestra.run.smithi181.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 79 kB/s | 45 kB 00:00 2024-09-21T21:26:48.232 INFO:teuthology.orchestra.run.smithi181.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 377 kB/s | 292 kB 00:00 2024-09-21T21:26:48.273 INFO:teuthology.orchestra.run.smithi181.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 756 kB/s | 662 kB 00:00 2024-09-21T21:26:48.516 INFO:teuthology.orchestra.run.smithi181.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 71 kB/s | 42 kB 00:00 2024-09-21T21:26:48.866 INFO:teuthology.orchestra.run.smithi181.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-09-21T21:26:48.916 INFO:teuthology.orchestra.run.smithi181.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 6.1 MB/s | 308 kB 00:00 2024-09-21T21:26:48.983 INFO:teuthology.orchestra.run.smithi181.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 294 kB/s | 221 kB 00:00 2024-09-21T21:26:49.033 INFO:teuthology.orchestra.run.smithi181.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 524 kB/s | 26 kB 00:00 2024-09-21T21:26:49.067 INFO:teuthology.orchestra.run.smithi181.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T21:26:49.101 INFO:teuthology.orchestra.run.smithi181.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-21T21:26:49.126 INFO:teuthology.orchestra.run.smithi181.stdout:(27/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 79 kB/s | 47 kB 00:00 2024-09-21T21:26:49.218 INFO:teuthology.orchestra.run.smithi181.stdout:(28/33): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-21T21:26:49.251 INFO:teuthology.orchestra.run.smithi181.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.5 MB/s | 839 kB 00:00 2024-09-21T21:26:49.285 INFO:teuthology.orchestra.run.smithi181.stdout:(30/33): re2-20211101-20.el9.x86_64.rpm 1.2 MB/s | 191 kB 00:00 2024-09-21T21:26:49.344 INFO:teuthology.orchestra.run.smithi181.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-21T21:26:49.919 INFO:teuthology.orchestra.run.smithi181.stdout:(32/33): librados2-19.3.0-5092.g62da54b4.el9.x8 5.2 MB/s | 3.5 MB 00:00 2024-09-21T21:26:50.170 INFO:teuthology.orchestra.run.smithi181.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 4.3 MB/s | 3.8 MB 00:00 2024-09-21T21:26:50.175 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:26:50.175 INFO:teuthology.orchestra.run.smithi181.stdout:Total 11 MB/s | 60 MB 00:05 2024-09-21T21:26:50.294 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:26:50.331 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:26:50.331 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:26:50.997 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:26:50.998 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:26:52.494 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:26:52.997 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T21:26:53.071 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:26:53.163 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:26:53.261 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:26:53.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:26:53.353 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:26:53.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-21T21:26:53.427 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-21T21:26:53.453 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-21T21:26:53.475 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:26:53.505 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:26:53.700 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:26:53.742 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:26:53.777 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-21T21:26:53.828 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-21T21:26:53.858 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:26:53.940 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-21T21:26:53.963 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:26:54.035 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:26:54.079 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-21T21:26:54.189 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-21T21:26:54.230 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-21T21:26:54.272 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-21T21:26:54.320 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-21T21:26:54.378 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-21T21:26:54.424 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-21T21:26:54.804 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T21:26:54.853 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:26:54.945 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:26:54.975 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-21T21:26:55.011 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T21:26:55.600 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T21:26:55.623 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:26:55.694 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:26:55.776 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-21T21:26:55.834 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-21T21:26:58.352 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:26:58.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:26:59.109 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:26:59.143 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:26:59.241 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:26:59.241 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T21:26:59.241 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T21:26:59.241 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:26:59.270 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-09-21T21:27:09.769 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-09-21T21:27:09.770 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-09-21T21:27:09.770 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-09-21T21:27:09.770 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:10.887 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:27:10.912 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:27:10.913 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:27:10.913 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T21:27:10.913 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:27:10.913 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:27:10.913 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:10.915 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:27:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:27:10.998 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:27:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:27:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-21T21:27:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:27:12.426 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T21:27:12.427 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T21:27:12.428 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T21:27:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:27:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T21:27:12.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.433 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.434 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T21:27:13.435 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T21:27:13.436 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:13.436 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:27:13.802 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-09-21T21:27:14.392 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:32 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:27:14.528 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:27:14.529 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-21T21:27:14.530 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:Install 9 Packages 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 83 M 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 775 M 2024-09-21T21:27:14.531 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:27:15.436 INFO:teuthology.orchestra.run.smithi181.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 589 kB/s | 94 kB 00:00 2024-09-21T21:27:15.503 INFO:teuthology.orchestra.run.smithi181.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 772 kB/s | 175 kB 00:00 2024-09-21T21:27:16.103 INFO:teuthology.orchestra.run.smithi181.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 304 kB/s | 183 kB 00:00 2024-09-21T21:27:16.212 INFO:teuthology.orchestra.run.smithi181.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 662 kB/s | 513 kB 00:00 2024-09-21T21:27:16.412 INFO:teuthology.orchestra.run.smithi181.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T21:27:16.487 INFO:teuthology.orchestra.run.smithi181.stdout:(6/9): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-09-21T21:27:16.521 INFO:teuthology.orchestra.run.smithi181.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 589 kB/s | 64 kB 00:00 2024-09-21T21:27:16.555 INFO:teuthology.orchestra.run.smithi181.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.8 MB/s | 122 kB 00:00 2024-09-21T21:27:18.088 INFO:teuthology.orchestra.run.smithi181.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 29 MB/s | 81 MB 00:02 2024-09-21T21:27:18.092 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:27:18.092 INFO:teuthology.orchestra.run.smithi181.stdout:Total 23 MB/s | 83 MB 00:03 2024-09-21T21:27:18.175 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:27:18.198 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:27:18.198 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:27:18.793 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:27:18.793 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:27:19.658 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:27:19.728 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-21T21:27:19.775 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-21T21:27:19.827 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-21T21:27:19.868 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-21T21:27:19.909 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:27:19.933 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:27:20.003 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:27:20.042 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-21T21:27:20.069 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:27:31.019 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:27:31.112 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:27:37.210 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-21T21:27:37.211 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-21T21:27:37.684 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-21T21:27:37.684 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:37.684 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:27:37.684 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:37.684 INFO:teuthology.orchestra.run.smithi181.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T21:27:37.685 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T21:27:37.686 INFO:teuthology.orchestra.run.smithi181.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-21T21:27:37.686 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:37.686 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:27:38.050 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-09-21T21:27:38.637 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:27:38.772 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:27:38.774 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:38.774 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-21T21:27:38.775 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T21:27:38.776 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T21:27:38.777 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T21:27:38.778 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T21:27:38.779 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T21:27:38.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout:Install 37 Packages 2024-09-21T21:27:38.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:38.782 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 84 M 2024-09-21T21:27:38.782 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 896 M 2024-09-21T21:27:38.783 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:27:39.198 INFO:teuthology.orchestra.run.smithi181.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 45 kB/s | 6.5 kB 00:00 2024-09-21T21:27:39.373 INFO:teuthology.orchestra.run.smithi181.stdout:(2/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 5.0 MB/s | 1.6 MB 00:00 2024-09-21T21:27:39.415 INFO:teuthology.orchestra.run.smithi181.stdout:(3/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 6.1 MB/s | 2.2 MB 00:00 2024-09-21T21:27:39.482 INFO:teuthology.orchestra.run.smithi181.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 4.3 MB/s | 292 kB 00:00 2024-09-21T21:27:39.615 INFO:teuthology.orchestra.run.smithi181.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2024-09-21T21:27:39.716 INFO:teuthology.orchestra.run.smithi181.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 12 MB/s | 1.2 MB 00:00 2024-09-21T21:27:39.783 INFO:teuthology.orchestra.run.smithi181.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-09-21T21:27:39.850 INFO:teuthology.orchestra.run.smithi181.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-21T21:27:39.933 INFO:teuthology.orchestra.run.smithi181.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.5 MB/s | 125 kB 00:00 2024-09-21T21:27:40.018 INFO:teuthology.orchestra.run.smithi181.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-21T21:27:40.109 INFO:teuthology.orchestra.run.smithi181.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 2.0 MB/s | 188 kB 00:00 2024-09-21T21:27:40.176 INFO:teuthology.orchestra.run.smithi181.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-21T21:27:40.259 INFO:teuthology.orchestra.run.smithi181.stdout:(13/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86 4.8 MB/s | 5.1 MB 00:01 2024-09-21T21:27:40.286 INFO:teuthology.orchestra.run.smithi181.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 319 kB/s | 35 kB 00:00 2024-09-21T21:27:40.360 INFO:teuthology.orchestra.run.smithi181.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 690 kB/s | 51 kB 00:00 2024-09-21T21:27:40.403 INFO:teuthology.orchestra.run.smithi181.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 538 kB/s | 77 kB 00:00 2024-09-21T21:27:40.428 INFO:teuthology.orchestra.run.smithi181.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 613 kB/s | 42 kB 00:00 2024-09-21T21:27:40.479 INFO:teuthology.orchestra.run.smithi181.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-21T21:27:40.504 INFO:teuthology.orchestra.run.smithi181.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-21T21:27:40.538 INFO:teuthology.orchestra.run.smithi181.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-09-21T21:27:40.563 INFO:teuthology.orchestra.run.smithi181.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 139 kB/s | 22 kB 00:00 2024-09-21T21:27:40.605 INFO:teuthology.orchestra.run.smithi181.stdout:(22/37): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-09-21T21:27:40.631 INFO:teuthology.orchestra.run.smithi181.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-09-21T21:27:40.656 INFO:teuthology.orchestra.run.smithi181.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-21T21:27:40.681 INFO:teuthology.orchestra.run.smithi181.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-21T21:27:40.706 INFO:teuthology.orchestra.run.smithi181.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-09-21T21:27:40.732 INFO:teuthology.orchestra.run.smithi181.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T21:27:40.757 INFO:teuthology.orchestra.run.smithi181.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 917 kB/s | 46 kB 00:00 2024-09-21T21:27:40.782 INFO:teuthology.orchestra.run.smithi181.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T21:27:40.816 INFO:teuthology.orchestra.run.smithi181.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 493 kB/s | 16 kB 00:00 2024-09-21T21:27:40.841 INFO:teuthology.orchestra.run.smithi181.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-21T21:27:40.867 INFO:teuthology.orchestra.run.smithi181.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T21:27:40.892 INFO:teuthology.orchestra.run.smithi181.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-21T21:27:40.917 INFO:teuthology.orchestra.run.smithi181.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-09-21T21:27:40.971 INFO:teuthology.orchestra.run.smithi181.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 377 kB/s | 20 kB 00:00 2024-09-21T21:27:41.001 INFO:teuthology.orchestra.run.smithi181.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T21:27:41.718 INFO:teuthology.orchestra.run.smithi181.stdout:(37/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 30 MB/s | 70 MB 00:02 2024-09-21T21:27:41.723 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:27:41.725 INFO:teuthology.orchestra.run.smithi181.stdout:Total 29 MB/s | 84 MB 00:02 2024-09-21T21:27:41.874 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:27:41.916 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:27:41.916 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:27:42.741 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:27:42.741 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:27:44.395 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:27:44.512 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T21:27:44.570 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T21:27:44.598 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T21:27:44.669 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T21:27:44.714 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T21:27:44.746 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T21:27:44.845 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T21:27:44.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T21:27:44.966 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T21:27:45.001 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T21:27:45.031 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T21:27:45.065 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T21:27:45.099 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T21:27:45.147 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T21:27:45.183 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T21:27:45.217 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T21:27:45.265 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T21:27:45.297 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T21:27:45.371 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T21:27:45.508 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T21:27:45.590 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T21:27:45.642 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T21:27:45.680 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T21:27:45.729 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T21:27:45.793 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T21:27:45.978 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T21:27:46.028 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T21:27:46.106 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T21:27:46.194 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T21:27:46.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T21:27:46.371 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T21:27:46.564 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-21T21:27:46.587 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:27:46.614 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:57.129 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:27:57.157 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:57.705 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:27:57.731 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:57.973 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:27:57.997 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:27:57.997 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:27:57.997 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T21:27:57.998 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:27:57.998 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:27:57.998 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:27:58.072 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T21:28:04.648 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T21:28:04.649 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T21:28:04.650 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T21:28:04.650 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T21:28:04.650 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T21:28:04.651 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:05.546 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T21:28:05.547 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T21:28:05.548 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T21:28:05.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T21:28:05.549 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T21:28:05.549 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:05.549 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:05.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-09-21T21:28:06.430 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:06.518 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:06.560 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:06.561 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:06.561 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:06.634 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-09-21T21:28:07.203 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:07.335 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:07.336 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:07.336 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:07.336 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:07.336 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 784 k 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 791 k 2024-09-21T21:28:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:07.622 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.7 MB/s | 784 kB 00:00 2024-09-21T21:28:07.623 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:07.623 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.7 MB/s | 784 kB 00:00 2024-09-21T21:28:07.624 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:07.629 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:07.629 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:07.635 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:07.635 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:07.706 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:08.168 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:08.260 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:08.613 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:08.864 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:08.864 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:08.864 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:08.864 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:08.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:08.865 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:09.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T21:28:09.698 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:09.830 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-21T21:28:09.831 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 158 k 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 644 k 2024-09-21T21:28:09.832 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:10.046 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 739 kB/s | 158 kB 00:00 2024-09-21T21:28:10.047 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:10.047 INFO:teuthology.orchestra.run.smithi181.stdout:Total 733 kB/s | 158 kB 00:00 2024-09-21T21:28:10.047 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:10.056 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:10.056 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:10.113 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:10.113 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:10.267 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:10.355 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:28:10.380 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:28:10.380 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:28:10.380 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T21:28:10.380 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:11.322 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:11.575 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-09-21T21:28:12.164 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:12.257 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:12.299 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:12.300 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:12.300 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:12.373 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T21:28:12.956 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:13.089 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:13.089 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T21:28:13.090 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout:Install 11 Packages 2024-09-21T21:28:13.091 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:13.092 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 18 M 2024-09-21T21:28:13.092 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 156 M 2024-09-21T21:28:13.092 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:13.825 INFO:teuthology.orchestra.run.smithi181.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 130 kB/s | 34 kB 00:00 2024-09-21T21:28:13.842 INFO:teuthology.orchestra.run.smithi181.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 59 kB/s | 16 kB 00:00 2024-09-21T21:28:13.984 INFO:teuthology.orchestra.run.smithi181.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-21T21:28:14.035 INFO:teuthology.orchestra.run.smithi181.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-09-21T21:28:14.076 INFO:teuthology.orchestra.run.smithi181.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 5.8 MB/s | 548 kB 00:00 2024-09-21T21:28:14.102 INFO:teuthology.orchestra.run.smithi181.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-09-21T21:28:14.144 INFO:teuthology.orchestra.run.smithi181.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-09-21T21:28:14.169 INFO:teuthology.orchestra.run.smithi181.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T21:28:14.211 INFO:teuthology.orchestra.run.smithi181.stdout:(9/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 15 MB/s | 2.0 MB 00:00 2024-09-21T21:28:14.244 INFO:teuthology.orchestra.run.smithi181.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-09-21T21:28:15.953 INFO:teuthology.orchestra.run.smithi181.stdout:(11/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 5.9 MB/s | 14 MB 00:02 2024-09-21T21:28:15.958 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:15.958 INFO:teuthology.orchestra.run.smithi181.stdout:Total 6.4 MB/s | 18 MB 00:02 2024-09-21T21:28:15.992 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:16.012 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:16.012 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:16.276 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:16.276 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:16.828 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:16.917 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-21T21:28:16.948 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-21T21:28:17.072 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-21T21:28:17.206 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-21T21:28:17.259 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-21T21:28:17.342 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-21T21:28:17.394 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-21T21:28:17.423 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:28:17.456 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-21T21:28:20.657 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-21T21:28:20.718 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-21T21:28:22.564 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-21T21:28:22.565 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-21T21:28:22.565 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:28:22.565 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-21T21:28:22.565 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-21T21:28:22.869 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:22.870 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:22.871 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:23.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T21:28:23.730 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:23.862 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T21:28:23.864 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-09-21T21:28:23.865 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:23.866 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 38 M 2024-09-21T21:28:23.866 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 212 M 2024-09-21T21:28:23.866 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:24.236 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 944 kB/s | 191 kB 00:00 2024-09-21T21:28:24.276 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.2 MB/s | 801 kB 00:00 2024-09-21T21:28:24.326 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 364 kB/s | 33 kB 00:00 2024-09-21T21:28:24.359 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 551 kB/s | 18 kB 00:00 2024-09-21T21:28:24.393 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 1.1 MB/s | 38 kB 00:00 2024-09-21T21:28:24.493 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-50 16 MB/s | 7.4 MB 00:00 2024-09-21T21:28:24.627 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-09-21T21:28:25.003 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.1 MB/s | 3.0 MB 00:00 2024-09-21T21:28:25.128 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-09-21T21:28:25.253 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.7 MB/s | 4.9 MB 00:00 2024-09-21T21:28:25.337 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.2 MB/s | 5.1 MB 00:00 2024-09-21T21:28:25.687 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-09-21T21:28:25.688 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:25.688 INFO:teuthology.orchestra.run.smithi181.stdout:Total 21 MB/s | 38 MB 00:01 2024-09-21T21:28:26.044 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:26.071 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:26.071 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:26.496 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:26.496 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:27.267 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:27.352 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T21:28:27.393 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:28:27.517 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:28:27.553 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T21:28:27.738 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T21:28:27.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T21:28:27.844 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T21:28:28.497 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T21:28:28.615 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T21:28:29.854 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:28:30.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T21:28:30.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:28:33.575 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:28:33.576 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T21:28:34.030 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T21:28:34.031 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T21:28:34.032 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T21:28:34.032 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T21:28:34.032 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:34.032 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:34.460 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-09-21T21:28:35.056 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:53 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:35.189 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-21T21:28:35.190 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:35.191 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:35.192 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-09-21T21:28:35.192 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:35.192 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.2 M 2024-09-21T21:28:35.192 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-09-21T21:28:35.192 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:35.469 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 211 kB/s | 19 kB 00:00 2024-09-21T21:28:35.494 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 26 kB 00:00 2024-09-21T21:28:35.553 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 282 kB/s | 49 kB 00:00 2024-09-21T21:28:35.578 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-21T21:28:35.603 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-21T21:28:35.628 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-21T21:28:35.670 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T21:28:35.696 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 233 kB/s | 14 kB 00:00 2024-09-21T21:28:35.729 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-21T21:28:35.763 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-21T21:28:35.788 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-21T21:28:35.813 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-21T21:28:35.847 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T21:28:35.849 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:35.849 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.4 MB/s | 2.2 MB 00:00 2024-09-21T21:28:35.913 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:35.928 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:35.929 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:36.020 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:36.026 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:36.532 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:36.710 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T21:28:36.766 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T21:28:36.820 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T21:28:36.857 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T21:28:36.899 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T21:28:37.003 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T21:28:37.111 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T21:28:37.159 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T21:28:37.889 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T21:28:37.931 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T21:28:37.965 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T21:28:38.013 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T21:28:38.043 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:28:38.862 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:28:38.862 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-21T21:28:38.862 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T21:28:38.863 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T21:28:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T21:28:39.201 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T21:28:39.202 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T21:28:39.203 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:39.203 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:39.470 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T21:28:40.080 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:58 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:40.212 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:40.213 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 7.0 M 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-09-21T21:28:40.214 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:40.766 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 725 kB/s | 154 kB 00:00 2024-09-21T21:28:40.858 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.8 MB/s | 548 kB 00:00 2024-09-21T21:28:40.900 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-21T21:28:40.933 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-21T21:28:41.325 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 322 kB/s | 248 kB 00:00 2024-09-21T21:28:41.700 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-21T21:28:41.702 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:41.703 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.7 MB/s | 7.0 MB 00:01 2024-09-21T21:28:41.774 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:41.785 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:41.786 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:41.900 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:41.900 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:42.213 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:42.385 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T21:28:42.436 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T21:28:42.953 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T21:28:43.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T21:28:43.106 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T21:28:43.131 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T21:28:43.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T21:28:44.171 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T21:28:44.171 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:44.171 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:44.172 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:44.443 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-09-21T21:28:45.026 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:03 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:45.158 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-09-21T21:28:45.159 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:45.160 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 963 k 2024-09-21T21:28:45.160 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 4.1 M 2024-09-21T21:28:45.160 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:45.435 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 523 kB/s | 80 kB 00:00 2024-09-21T21:28:45.569 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.0 MB/s | 883 kB 00:00 2024-09-21T21:28:45.569 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:45.570 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.3 MB/s | 963 kB 00:00 2024-09-21T21:28:45.581 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:45.604 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:45.605 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:45.686 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:45.686 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:45.888 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:46.038 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T21:28:46.110 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:28:46.758 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:28:46.758 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:47.014 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:47.352 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-09-21T21:28:47.951 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:05 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:48.089 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:48.089 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:48.089 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:28:48.089 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:28:48.090 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:48.091 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 285 k 2024-09-21T21:28:48.091 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.3 M 2024-09-21T21:28:48.091 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:48.324 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T21:28:48.324 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:48.324 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T21:28:48.325 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:48.333 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:48.333 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:48.359 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:48.360 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:48.588 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:48.643 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:48.658 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:48.659 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:28:48.659 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T21:28:48.659 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:49.667 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:50.004 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-09-21T21:28:50.602 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:08 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:50.740 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:50.742 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 125 k 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 445 k 2024-09-21T21:28:50.743 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:50.954 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 588 kB/s | 125 kB 00:00 2024-09-21T21:28:50.955 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:50.956 INFO:teuthology.orchestra.run.smithi181.stdout:Total 584 kB/s | 125 kB 00:00 2024-09-21T21:28:50.956 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:50.960 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:50.961 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:51.024 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:51.024 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:51.177 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:51.277 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:51.692 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:51.949 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:52.211 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-09-21T21:28:52.810 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:10 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:52.901 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:52.946 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:52.947 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:52.947 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:53.029 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-09-21T21:28:53.610 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:11 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:53.742 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:53.743 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 32 k 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 142 k 2024-09-21T21:28:53.744 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:28:53.900 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 204 kB/s | 32 kB 00:00 2024-09-21T21:28:53.900 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:28:53.900 INFO:teuthology.orchestra.run.smithi181.stdout:Total 202 kB/s | 32 kB 00:00 2024-09-21T21:28:53.900 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:28:53.904 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:28:53.904 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:28:53.921 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:28:53.922 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:28:54.008 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:28:54.130 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:54.547 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:28:54.781 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:54.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-09-21T21:28:55.553 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:13 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:55.643 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:55.687 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:55.688 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:55.688 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:55.758 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-09-21T21:28:56.343 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:14 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:56.433 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:56.476 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:56.477 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:56.477 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:56.552 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-09-21T21:28:57.135 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:15 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:57.224 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:57.267 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:57.268 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:57.268 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:57.343 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-09-21T21:28:57.919 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:15 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:58.008 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:58.051 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:58.052 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:58.052 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:58.122 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-09-21T21:28:58.708 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:16 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:58.798 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:58.841 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:58.842 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:58.842 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:58.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-09-21T21:28:59.505 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:17 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:28:59.593 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:28:59.637 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:28:59.638 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-09-21T21:28:59.638 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:28:59.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-09-21T21:29:00.303 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:18 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:29:00.444 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 91 k 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 321 k 2024-09-21T21:29:00.445 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:29:00.646 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 456 kB/s | 91 kB 00:00 2024-09-21T21:29:00.646 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:00.646 INFO:teuthology.orchestra.run.smithi181.stdout:Total 452 kB/s | 91 kB 00:00 2024-09-21T21:29:00.646 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:29:00.655 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:29:00.655 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:29:00.722 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:29:00.722 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:29:00.878 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:29:01.003 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:01.396 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:01.665 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:29:01.970 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-09-21T21:29:02.564 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:20 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:29:02.702 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:02.703 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:29:02.704 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:02.704 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.5 M 2024-09-21T21:29:02.704 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 30 M 2024-09-21T21:29:02.704 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:29:03.058 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 9.9 MB/s | 3.5 MB 00:00 2024-09-21T21:29:03.058 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:03.059 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.9 MB/s | 3.5 MB 00:00 2024-09-21T21:29:03.059 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:29:03.070 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:29:03.070 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:29:03.155 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:29:03.156 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:29:03.739 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:29:03.817 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:03.842 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:29:03.843 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T21:29:03.843 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:29:03.843 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:29:03.843 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:05.087 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:05.087 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:05.087 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:29:05.087 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:05.087 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:05.088 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:29:05.355 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-09-21T21:29:05.962 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:23 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:29:06.097 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:06.098 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 176 k 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 685 k 2024-09-21T21:29:06.099 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:29:06.287 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 940 kB/s | 176 kB 00:00 2024-09-21T21:29:06.287 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:06.287 INFO:teuthology.orchestra.run.smithi181.stdout:Total 932 kB/s | 176 kB 00:00 2024-09-21T21:29:06.288 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:29:06.297 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:29:06.297 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:29:06.363 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:29:06.363 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:29:06.539 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:29:06.642 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:07.067 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:07.336 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:29:07.336 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:07.336 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:29:07.337 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:07.337 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:07.337 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:29:07.625 DEBUG:teuthology.parallel:result is None 2024-09-21T21:29:07.626 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:29:07.762 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T21:29:07.784 INFO:teuthology.orchestra.run.smithi181.stdout:19.3.0-5092.g62da54b4.el9 2024-09-21T21:29:07.784 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-21T21:29:07.784 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-21T21:29:07.786 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T21:29:07.786 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:07.786 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T21:29:07.850 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T21:29:07.851 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:07.851 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T21:29:07.915 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T21:29:07.979 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T21:29:07.980 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:07.980 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T21:29:08.044 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T21:29:08.109 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T21:29:08.109 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:08.109 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T21:29:08.174 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T21:29:08.241 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T21:29:08.315 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T21:29:08.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-09-21T21:29:08.348 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T21:29:08.348 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T21:29:08.413 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T21:29:08.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T21:29:08.485 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T21:29:08.486 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T21:29:08.486 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4K_rand_read}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914348', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOk8KP3XpemVFVE2VMkzXmkzVvwLe/vw8QtLyjjkejqHYqoT8AK605BMuMP2Cb5X0dffaypgvtlHQuKX3DISWlQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T21:29:08.486 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T21:29:08.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T21:29:08.600 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:08.600 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T21:29:08.656 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T21:29:08.656 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-09-21T21:29:08.711 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T21:29:08.711 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:29:08.711 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T21:29:08.711 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:29:08.712 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:29:08.712 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 21:29:04.107037767 +0000 2024-09-21T21:29:08.712 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 21:26:06.151805248 +0000 2024-09-21T21:29:08.712 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 21:26:06.151805248 +0000 2024-09-21T21:29:08.712 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 21:26:06.151805248 +0000 2024-09-21T21:29:08.712 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T21:29:08.774 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T21:29:08.774 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T21:29:08.774 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000240399 s, 2.1 MB/s 2024-09-21T21:29:08.775 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T21:29:08.831 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-09-21T21:29:08.886 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T21:29:08.886 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:29:08.886 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 21:29:04.107037767 +0000 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 21:26:06.467798942 +0000 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 21:26:06.467798942 +0000 2024-09-21T21:29:08.887 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 21:26:06.467798942 +0000 2024-09-21T21:29:08.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T21:29:08.950 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T21:29:08.951 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T21:29:08.951 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000234084 s, 2.2 MB/s 2024-09-21T21:29:08.952 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T21:29:09.009 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-09-21T21:29:09.066 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T21:29:09.066 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:29:09.066 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-21T21:29:09.066 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:29:09.066 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:29:09.067 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 21:29:04.107037767 +0000 2024-09-21T21:29:09.067 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 21:26:06.782792656 +0000 2024-09-21T21:29:09.067 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 21:26:06.782792656 +0000 2024-09-21T21:29:09.067 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 21:26:06.782792656 +0000 2024-09-21T21:29:09.067 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T21:29:09.129 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T21:29:09.129 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T21:29:09.129 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000155826 s, 3.3 MB/s 2024-09-21T21:29:09.131 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T21:29:09.187 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-09-21T21:29:09.243 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 726 Links: 1 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-09-21 21:29:04.108037747 +0000 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-09-21 21:26:07.088786549 +0000 2024-09-21T21:29:09.244 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-09-21 21:26:07.088786549 +0000 2024-09-21T21:29:09.245 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-09-21 21:26:07.088786549 +0000 2024-09-21T21:29:09.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T21:29:09.306 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-09-21T21:29:09.307 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-09-21T21:29:09.307 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000185547 s, 2.8 MB/s 2024-09-21T21:29:09.308 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T21:29:09.365 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:29:09.365 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:29:09.365 INFO:tasks.ceph:Generating config... 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-21T21:29:09.366 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-21T21:29:09.367 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T21:29:09.367 INFO:tasks.ceph:[global] auth client required = none 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] auth service required = none 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] auth supported = none 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-21T21:29:09.368 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-21T21:29:09.369 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-21T21:29:09.370 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-21T21:29:09.371 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-21T21:29:09.372 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-21T21:29:09.373 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-21T21:29:09.374 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-21T21:29:09.375 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-21T21:29:09.376 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-21T21:29:09.376 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-21T21:29:09.377 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-21T21:29:09.377 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-21T21:29:09.377 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-21T21:29:09.377 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T21:29:09.377 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T21:29:09.464 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T21:29:09.467 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T21:29:09.519 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:29:09.543 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.181')] 2024-09-21T21:29:09.544 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.181', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-21T21:29:09.545 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:09.545 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:29:09.599 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:09.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:09.688+0000 7f3c7b835f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:09.691 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:09.689+0000 7f3c7b835f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:09.692 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:09.689+0000 7f3c7b835f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:09.692 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:09.692 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: generated fsid 32057cd2-c78d-44db-a892-76c556c7a276 2024-09-21T21:29:09.692 INFO:teuthology.orchestra.run.smithi181.stdout:setting min_mon_release = quincy 2024-09-21T21:29:09.692 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 0 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:fsid 32057cd2-c78d-44db-a892-76c556c7a276 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:last_changed 2024-09-21T21:29:09.691339+0000 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-09-21T21:29:09.691339+0000 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:min_mon_release 17 (quincy) 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:election_strategy: 1 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:0: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.a 2024-09-21T21:29:09.693 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-21T21:29:09.694 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:29:09.709 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 32057cd2-c78d-44db-a892-76c556c7a276... 2024-09-21T21:29:09.710 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = none 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.792 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T21:29:09.793 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-09-21T21:29:09.794 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = 172.21.15.181 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: auth client required = none 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: auth cluster required = none 2024-09-21T21:29:09.795 INFO:teuthology.orchestra.run.smithi181.stdout: auth service required = none 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug asok = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug bdev = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug buffer = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug civetweb = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug client = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug compressor = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug context = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug crush = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug crypto = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug dpdk = 0/0 2024-09-21T21:29:09.796 INFO:teuthology.orchestra.run.smithi181.stdout: debug eventtrace = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug filer = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug filestore = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug finisher = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug fuse = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug heartbeatmap = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug javaclient = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug journal = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug journaler = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug kstore = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug leveldb = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug lockdep = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds balancer = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds locker = 0/0 2024-09-21T21:29:09.797 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds log = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds log expire = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds migrator = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug memdb = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgrc = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug objclass = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug objectcacher = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug objecter = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug optracker = 0/0 2024-09-21T21:29:09.798 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug perfcounter = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rados = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd mirror = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd replay = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug refs = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw lifecycle = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw notification = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw sync = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug striper = 0/0 2024-09-21T21:29:09.799 INFO:teuthology.orchestra.run.smithi181.stdout: debug throttle = 0/0 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: debug timer = 0/0 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: debug tp = 0/0 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms client mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms cluster mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon client mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon cluster mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon service mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: ms service mode = crc 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = 32057cd2-c78d-44db-a892-76c556c7a276 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-09-21T21:29:09.800 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-09-21T21:29:09.801 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-09-21T21:29:09.802 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: crimson alien thread cpu cores = 6-7 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: crimson osd obc lru size = 10 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug filestore = 0/0 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug journal = 0/0 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 0/0 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore = 20 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore backref = 20 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore cache = 20 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore cleaner = 20 2024-09-21T21:29:09.803 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore device = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore epm = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore fixedkv tree = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore journal = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore lba = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore odata = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore ompap = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore onode = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore t = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug seastore tm = 20 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = seastore 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-09-21T21:29:09.804 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 0/0 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 0/0 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-09-21T21:29:09.805 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.806 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default crimson = True 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:29:09.807 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-21T21:29:09.808 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:29:09.808 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-21T21:29:09.808 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:29:09.808 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.3] 2024-09-21T21:29:09.808 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:29:09.808 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T21:29:09.809 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T21:29:09.913 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T21:29:09.913 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:09.913 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T21:29:09.929 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:09.929 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T21:29:09.984 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:29:09.985 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:09.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T21:29:09.985 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:29:10.058 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:10.058 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:10.114 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T21:29:10.114 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T21:29:10.114 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:29:10.213 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:29:10.216 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T21:29:10.216 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T21:29:10.216 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T21:29:10.265 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T21:29:10.278 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T21:29:10.279 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:29:10.279 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T21:29:10.303 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:29:10.303 INFO:tasks.ceph:role: osd.0 2024-09-21T21:29:10.303 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:29:10.303 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T21:29:10.375 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:29:10.375 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:29:10.375 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:29:10.375 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:29:10.376 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:29:10.889 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-21T21:29:10.890 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-21T21:29:10.891 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T21:29:11.031 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T21:29:11.064 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T21:29:11.130 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:29:11.130 INFO:tasks.ceph:role: osd.1 2024-09-21T21:29:11.130 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:29:11.130 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:29:11.203 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:29:11.656 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-21T21:29:11.658 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-21T21:29:11.659 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T21:29:11.750 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T21:29:11.778 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T21:29:11.843 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:29:11.843 INFO:tasks.ceph:role: osd.2 2024-09-21T21:29:11.843 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:29:11.843 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:29:11.915 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:29:11.916 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:29:11.916 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:29:11.916 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:29:12.406 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-09-21T21:29:12.408 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-09-21T21:29:12.408 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T21:29:12.559 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T21:29:12.590 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:12.886 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:12.887 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:12,887 seastar - Reactor backend: linux-aio 2024-09-21T21:29:12.898 INFO:teuthology.orchestra.run.smithi181.stderr:==27891==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:13.028 INFO:teuthology.orchestra.run.smithi181.stderr:DEBUG 2024-09-21 21:29:13,027 [shard 0:main] seastar - reactor::drain 2024-09-21T21:29:13.550 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:13.550 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:13.551 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:13.551 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:13.551 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:13.551 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:13.604 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:13.608 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:13,607 seastar - Reactor backend: linux-aio 2024-09-21T21:29:13.617 INFO:teuthology.orchestra.run.smithi181.stderr:==27889==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:13.904 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3816bb1 0x3ac72ab 0x3b8c153 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9d02c7 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:13.904 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3610a40 0x3610d67 0xba0be37 0xba0c16c 0xba0c21c 0xba0c747 0xba0d1d1 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:13.904 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:13.904 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:13.961 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e5b961 0x9e5c1d8 0x9e5c8fd 0x9e04b13 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:13.962 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:13.962 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4b865 0x7496ebf 0x7497da0 0x9eb9447 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:13.962 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.169 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55250b4 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:14.169 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.227 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524e7d 0x5525e58 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:14.227 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.247 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cf0 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.247 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.247 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525829 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.247 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.305 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab7c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.305 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.305 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab17 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.305 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.334 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cde 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:14.334 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.610 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x377957f 0x3ac7299 0x3b8c153 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9e567f 0xba14771 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.610 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x124f7c 0x421f3 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb836d56 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xba13cb0 0xba13f4c 0xba14113 0xba14387 0xba14490 0xba1460c 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:14.611 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.611 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.745 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ea8 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:14.745 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:14.802 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16cf4 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:14.803 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:15.705 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:15.706 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:15.706 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:15.706 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:15.706 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:15.706 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:15.732 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T21:29:15.757 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:16.038 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:16.039 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:16,039 seastar - Reactor backend: linux-aio 2024-09-21T21:29:16.045 INFO:teuthology.orchestra.run.smithi181.stderr:==27938==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:16.174 INFO:teuthology.orchestra.run.smithi181.stderr:DEBUG 2024-09-21 21:29:16,173 [shard 0:main] seastar - reactor::drain 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:16.672 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:16.728 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:16.732 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:16,731 seastar - Reactor backend: linux-aio 2024-09-21T21:29:16.748 INFO:teuthology.orchestra.run.smithi181.stderr:==27936==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:16.813 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3ad3f 0x10d76c 0xdd31a 0x4b14c 0x381a3f4 0x5fc081e 0x5fc09b5 0x601bbd4 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:16.814 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.034 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab9d 0x3ac7302 0x3b8c153 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.034 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.034 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41a2a 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0xb83682d 0xba16ccc 0xba16dfe 0xba16f7a 0xba17729 0xb9b1eb4 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.034 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.092 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x162bff 0x422c4 0x3ef7b 0xfd531 0x3c34e46 0x553c478 0x9eb429d 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.092 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3643d17 0x3643d7b 0x3643e37 0x3644204 0x36e0ceb 0x36e11ca 0x36e1614 0xc4a96ec 0xc49545e 0x9e20876 0x9df83e9 0x9e054a3 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.092 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.092 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff932a5c77 0xffffffff92fb9f96 0xffffffff92fbdc41 0xffffffff92fbeda2 0xffffffff92fbf617 0xffffffff92fe47b3 0xffffffff92f8eb3e 0xffffffff92f9699b 0xffffffff92f96e1c 0xffffffff92c8c884 0xffffffff938d1db2 0xffffffff93a00bb2 2024-09-21T21:29:17.283 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b581 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:17.283 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.340 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525fe5 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:17.341 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.373 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133caf 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.373 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.373 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133ccf 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.373 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.430 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525e08 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.430 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.431 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16ce8 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.431 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.448 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab5b 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:17.448 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.746 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x961a149 0x9add440 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.746 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.746 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c23 0xdd332 0x4ab1c 0xba0d019 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:17.746 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:17.880 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525898 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:17.880 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:18.830 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:18.830 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:18.830 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:18.830 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:18.831 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:18.831 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:18.856 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T21:29:18.881 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:19.167 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:19.168 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:19,167 seastar - Reactor backend: linux-aio 2024-09-21T21:29:19.174 INFO:teuthology.orchestra.run.smithi181.stderr:==27985==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:19.280 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x361a524 0x36292ab 0x382a100 0x416e695 0x5f9bd94 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:19.280 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:19.302 INFO:teuthology.orchestra.run.smithi181.stderr:DEBUG 2024-09-21 21:29:19,301 [shard 0:main] seastar - reactor::drain 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:19.805 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:19.862 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:29:19.866 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-09-21 21:29:19,865 seastar - Reactor backend: linux-aio 2024-09-21T21:29:19.882 INFO:teuthology.orchestra.run.smithi181.stderr:==27983==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:29:20.165 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3610a40 0x3610d67 0xba0be37 0xba0c16c 0xba0c21c 0xba0c747 0xba0d1d1 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.165 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.166 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x162c49 0x422c4 0x3ef7b 0xfd531 0xb9f4562 0xba0c4c1 0xba0d1d1 0xb9bfd1b 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.166 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff92c8c7a4 0xffffffff938d1db2 0xffffffff93a00bb2 2024-09-21T21:29:20.223 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8cc70 0x360b57f 0x36e0ef3 0x36e11ca 0x36e1614 0xb0bceba 0xb0bd20c 0xb0bd36c 0xc494f32 0xc495542 0xb0b6299 0xb0b0970 0xac78421 0xacde0f1 0xacab3b6 0x99296ea 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.223 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.223 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x162c49 0x4162a 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ecea 0xfc9b1 0x3869141 0xb9f36a3 0xba0b16e 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9ae1c41 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.223 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff93288f80 0xffffffff9328c7db 0xffffffff92f9b83c 0xffffffff92c8c811 0xffffffff938d1db2 0xffffffff93a00bb2 2024-09-21T21:29:20.224 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3829f48 0x455104a 0xb9b550a 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:20.224 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.282 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab09 0x3ac7302 0x3b8c153 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:20.282 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.346 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3868f40 0xb9b1e4b 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.346 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.413 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cbb 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:20.413 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.470 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10798a 0xdd2e1 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:20.471 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.516 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133caf 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.516 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.517 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c27 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.517 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.574 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524ea6 0x5525e58 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.574 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.575 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55250ab 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.575 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.578 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x552581d 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:29:20.578 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:20.681 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525fd6 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:29:20.681 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-09-21T21:29:21.972 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:22.000 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T21:29:22.025 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T21:29:22.026 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:22.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T21:29:22.090 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:22.091 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T21:29:22.155 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:22.155 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T21:29:22.220 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:22.220 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T21:29:22.285 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:22.285 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T21:29:22.340 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T21:29:22.340 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCmOu9mbd+oDBAAdZyxq2NRAXT+1xh4EjkwTA== 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCpOu9mEgxgLRAAhi7jM5idUacSJBnJuh8VOg== 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCsOu9m7sobNRAA1w6W1o38ZTEGMkLgo/UdVA== 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCwOu9mHdibARAARzmEuNjEzeV7OLxm7xnkWA== 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-09-21T21:29:22.404 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQCmOu9mrUHPDxAACO4WeYGxmUfuRM+Z5zaOgQ== 2024-09-21T21:29:22.405 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T21:29:22.506 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:29:22.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:29:22.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:29:22.699 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T21:29:22.763 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T21:29:22.764 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T21:29:22.788 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T21:29:22.903 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:22.902+0000 7f8c5951bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:22.903 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:22.902+0000 7f8c5951bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:22.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T21:29:22.942 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:29:22.997 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T21:29:22.998 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T21:29:22.998 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T21:29:23.040 INFO:tasks.ceph.mon.a:Started 2024-09-21T21:29:23.040 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T21:29:23.040 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T21:29:23.041 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T21:29:23.042 INFO:tasks.ceph.mgr.x:Started 2024-09-21T21:29:23.043 DEBUG:tasks.ceph:set 0 configs 2024-09-21T21:29:23.043 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T21:29:23.043 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T21:29:23.124 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-21T21:29:23.122+0000 7f6c84a3bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.124 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-21T21:29:23.122+0000 7f6c84a3bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.124 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-21T21:29:23.122+0000 7f6c84a3bc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.124 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-21T21:29:23.123+0000 7f650fa32000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.153 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:23.146+0000 7f855f66c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.155 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:23.153+0000 7f855f66c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.223 INFO:teuthology.orchestra.run.smithi181.stderr:adjusted tunables profile to default 2024-09-21T21:29:23.235 INFO:tasks.ceph:check_enable_crimson: True 2024-09-21T21:29:23.235 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-21T21:29:23.235 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-21T21:29:23.329 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-21T21:29:23.327+0000 7f650fa32000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.329 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-21T21:29:23.328+0000 7f650fa32000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.341 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:23.335+0000 7f4b7a191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.343 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:23.342+0000 7f4b7a191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:23.443 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-09-21T21:29:23.442+0000 7f6c7b3f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-21T21:29:23.444 INFO:tasks.ceph.mon.a.smithi181.stderr:Please be aware that this feature is experimental, untested, 2024-09-21T21:29:23.444 INFO:tasks.ceph.mon.a.smithi181.stderr:unsupported, and may result in data corruption, data loss, 2024-09-21T21:29:23.444 INFO:tasks.ceph.mon.a.smithi181.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-21T21:29:23.444 INFO:tasks.ceph.mon.a.smithi181.stderr:feature with important data. 2024-09-21T21:29:23.444 INFO:tasks.ceph.mon.a.smithi181.stderr: 2024-09-21T21:29:24.235 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T21:29:24.236 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:24.236 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T21:29:24.260 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:24.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T21:29:24.325 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:24.326 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T21:29:24.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9 0 2024-09-21T21:29:24.539 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.532+0000 7f1b0bd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:24.541 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.540+0000 7f1b0bd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:24.617 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-09-21T21:29:24.629 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 5d0eeaeb-6063-4b08-9118-2cf8b00c84ca 1 2024-09-21T21:29:24.737 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.729+0000 7fdeeab91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:24.739 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.737+0000 7fdeeab91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:24.815 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-09-21T21:29:24.827 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 5c7f65da-6b48-40f9-9647-393373829e82 2 2024-09-21T21:29:24.936 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.928+0000 7ff2f8308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:24.938 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:24.936+0000 7ff2f8308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:25.014 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-09-21T21:29:25.026 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-21T21:29:25.055 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T21:29:25.055 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-21T21:29:25.099 INFO:tasks.ceph.osd.0:Started 2024-09-21T21:29:25.099 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-21T21:29:25.130 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T21:29:25.130 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-21T21:29:25.172 INFO:tasks.ceph.osd.1:Started 2024-09-21T21:29:25.172 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-21T21:29:25.202 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T21:29:25.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-21T21:29:25.245 INFO:tasks.ceph.osd.2:Started 2024-09-21T21:29:25.245 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:29:25.387 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:25.382+0000 7fe913f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:25.390 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:25.389+0000 7fe913f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:25.458 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:25.458 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":6,"fsid":"32057cd2-c78d-44db-a892-76c556c7a276","created":"2024-09-21T21:29:23.143576+0000","modified":"2024-09-21T21:29:25.012943+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:29:25.012943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5d0eeaeb-6063-4b08-9118-2cf8b00c84ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c7f65da-6b48-40f9-9647-393373829e82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:29:25.470 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T21:29:25.470 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T21:29:25.772 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:29:25.876 INFO:teuthology.misc.health.smithi181.stderr:2024-09-21T21:29:25.870+0000 7f3feed51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:25.878 INFO:teuthology.misc.health.smithi181.stderr:2024-09-21T21:29:25.877+0000 7f3feed51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:25.948 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-21T21:29:25.948 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":6,"fsid":"32057cd2-c78d-44db-a892-76c556c7a276","created":"2024-09-21T21:29:23.143576+0000","modified":"2024-09-21T21:29:25.012943+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:29:25.012943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5d0eeaeb-6063-4b08-9118-2cf8b00c84ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5c7f65da-6b48-40f9-9647-393373829e82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:29:25.957 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T21:29:28.875 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-09-21T21:29:28.873+0000 7f64ac95c640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T21:29:32.260 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:29:32.371 INFO:teuthology.misc.health.smithi181.stderr:2024-09-21T21:29:32.366+0000 7f48b8591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:32.375 INFO:teuthology.misc.health.smithi181.stderr:2024-09-21T21:29:32.373+0000 7f48b8591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:32.580 INFO:teuthology.misc.health.smithi181.stdout: 2024-09-21T21:29:32.580 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":10,"fsid":"32057cd2-c78d-44db-a892-76c556c7a276","created":"2024-09-21T21:29:23.143576+0000","modified":"2024-09-21T21:29:31.929795+0000","last_up_change":"2024-09-21T21:29:31.929795+0000","last_in_change":"2024-09-21T21:29:25.012943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2515308370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2515308370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2515308370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2515308370}]},"public_addr":"172.21.15.181:6805/2515308370","cluster_addr":"172.21.15.181:6804/2515308370","heartbeat_back_addr":"172.21.15.181:6814/2515308370","heartbeat_front_addr":"172.21.15.181:6813/2515308370","state":["exists","up"]},{"osd":1,"uuid":"5d0eeaeb-6063-4b08-9118-2cf8b00c84ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1230609635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1230609635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1230609635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1230609635}]},"public_addr":"172.21.15.181:6803/1230609635","cluster_addr":"172.21.15.181:6802/1230609635","heartbeat_back_addr":"172.21.15.181:6810/1230609635","heartbeat_front_addr":"172.21.15.181:6811/1230609635","state":["exists","up"]},{"osd":2,"uuid":"5c7f65da-6b48-40f9-9647-393373829e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":748213074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":748213074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":748213074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":748213074}]},"public_addr":"172.21.15.181:6807/748213074","cluster_addr":"172.21.15.181:6806/748213074","heartbeat_back_addr":"172.21.15.181:6815/748213074","heartbeat_front_addr":"172.21.15.181:6812/748213074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:29:32.594 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T21:29:32.594 INFO:tasks.ceph:Creating RBD pool 2024-09-21T21:29:32.594 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T21:29:32.715 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:32.705+0000 7f325e311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:32.718 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:32.716+0000 7f325e311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:32.943 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'rbd' created 2024-09-21T21:29:32.969 DEBUG:teuthology.orchestra.run.smithi181:> rbd --cluster ceph pool init rbd 2024-09-21T21:29:33.077 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:33.075+0000 7f5f3ea3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:33.077 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:33.075+0000 7f5f3ea3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:33.080 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:33.078+0000 7f5f3ea3eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:36.060 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T21:29:36.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T21:29:36.061 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T21:29:36.183 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:36.176+0000 7f04c44d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:36.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:36.184+0000 7f04c44d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:36.501 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:36.516 INFO:teuthology.orchestra.run.smithi181.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-21T21:29:36.517 INFO:tasks.ceph_manager:config epoch is 1 2024-09-21T21:29:36.517 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T21:29:36.517 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T21:29:36.517 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T21:29:36.637 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:36.632+0000 7f26cacac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:36.640 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:36.638+0000 7f26cacac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:36.888 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:36.900 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3676001273},{"type":"v1","addr":"172.21.15.181:6801","nonce":3676001273}]},"active_addr":"172.21.15.181:6801/3676001273","active_change":"2024-09-21T21:29:27.860208+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":4284235377}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":843352309}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":316962245}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":421972421}]}]} 2024-09-21T21:29:36.903 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T21:29:36.903 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T21:29:36.903 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:29:37.016 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.011+0000 7f27e0d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.018 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.016+0000 7f27e0d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.214 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:37.214 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":14,"fsid":"32057cd2-c78d-44db-a892-76c556c7a276","created":"2024-09-21T21:29:23.143576+0000","modified":"2024-09-21T21:29:35.953099+0000","last_up_change":"2024-09-21T21:29:31.929795+0000","last_in_change":"2024-09-21T21:29:25.012943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:29:32.926346+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:29:33.888312+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2515308370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2515308370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2515308370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2515308370}]},"public_addr":"172.21.15.181:6805/2515308370","cluster_addr":"172.21.15.181:6804/2515308370","heartbeat_back_addr":"172.21.15.181:6814/2515308370","heartbeat_front_addr":"172.21.15.181:6813/2515308370","state":["exists","up"]},{"osd":1,"uuid":"5d0eeaeb-6063-4b08-9118-2cf8b00c84ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1230609635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1230609635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1230609635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1230609635}]},"public_addr":"172.21.15.181:6803/1230609635","cluster_addr":"172.21.15.181:6802/1230609635","heartbeat_back_addr":"172.21.15.181:6810/1230609635","heartbeat_front_addr":"172.21.15.181:6811/1230609635","state":["exists","up"]},{"osd":2,"uuid":"5c7f65da-6b48-40f9-9647-393373829e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":748213074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":748213074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":748213074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":748213074}]},"public_addr":"172.21.15.181:6807/748213074","cluster_addr":"172.21.15.181:6806/748213074","heartbeat_back_addr":"172.21.15.181:6815/748213074","heartbeat_front_addr":"172.21.15.181:6812/748213074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:29:37.227 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T21:29:37.227 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:29:37.335 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.330+0000 7f6a44591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.337 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.336+0000 7f6a44591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:37.522 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":14,"fsid":"32057cd2-c78d-44db-a892-76c556c7a276","created":"2024-09-21T21:29:23.143576+0000","modified":"2024-09-21T21:29:35.953099+0000","last_up_change":"2024-09-21T21:29:31.929795+0000","last_in_change":"2024-09-21T21:29:25.012943+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:29:32.926346+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:29:33.888312+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f2daa3a9-b60b-4b65-bc13-3b0eef2c70d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":2515308370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":2515308370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":2515308370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2515308370}]},"public_addr":"172.21.15.181:6805/2515308370","cluster_addr":"172.21.15.181:6804/2515308370","heartbeat_back_addr":"172.21.15.181:6814/2515308370","heartbeat_front_addr":"172.21.15.181:6813/2515308370","state":["exists","up"]},{"osd":1,"uuid":"5d0eeaeb-6063-4b08-9118-2cf8b00c84ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1230609635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1230609635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":1230609635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":1230609635}]},"public_addr":"172.21.15.181:6803/1230609635","cluster_addr":"172.21.15.181:6802/1230609635","heartbeat_back_addr":"172.21.15.181:6810/1230609635","heartbeat_front_addr":"172.21.15.181:6811/1230609635","state":["exists","up"]},{"osd":2,"uuid":"5c7f65da-6b48-40f9-9647-393373829e82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":748213074}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":748213074}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6815","nonce":748213074}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":748213074}]},"public_addr":"172.21.15.181:6807/748213074","cluster_addr":"172.21.15.181:6806/748213074","heartbeat_back_addr":"172.21.15.181:6815/748213074","heartbeat_front_addr":"172.21.15.181:6812/748213074","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:29:37.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T21:29:37.535 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T21:29:37.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T21:29:37.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.645+0000 7f38a78d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.652 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.645+0000 7ff787d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.653 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.645+0000 7fc611391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.652+0000 7fc611391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.652+0000 7f38a78d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.654 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.653+0000 7ff787d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.682 INFO:teuthology.orchestra.run.smithi181.stdout:42949672965 2024-09-21T21:29:37.682 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:29:37.685 INFO:teuthology.orchestra.run.smithi181.stdout:34359738375 2024-09-21T21:29:37.685 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:29:37.687 INFO:teuthology.orchestra.run.smithi181.stdout:42949672965 2024-09-21T21:29:37.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:29:37.794 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.788+0000 7fbe49ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.797 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.795+0000 7fbe49ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.840 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.834+0000 7fc3684d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.841 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.835+0000 7f6cc7991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.843 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.841+0000 7fc3684d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:37.844 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:37.842+0000 7f6cc7991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:38.000 INFO:teuthology.orchestra.run.smithi181.stdout:42949672964 2024-09-21T21:29:38.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672964 for osd.0 2024-09-21T21:29:38.037 INFO:teuthology.orchestra.run.smithi181.stdout:42949672964 2024-09-21T21:29:38.047 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-09-21T21:29:38.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672964 for osd.2 2024-09-21T21:29:38.060 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738373 for osd.1 2024-09-21T21:29:39.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:29:39.051 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:29:39.060 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:29:39.143 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.137+0000 7f7c3365c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.146 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.144+0000 7f7c3365c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.209+0000 7f7e1b111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.216 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.209+0000 7f3811391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.216+0000 7f7e1b111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.218 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.216+0000 7f3811391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.355 INFO:teuthology.orchestra.run.smithi181.stdout:42949672965 2024-09-21T21:29:39.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.0 2024-09-21T21:29:39.369 DEBUG:teuthology.parallel:result is None 2024-09-21T21:29:39.420 INFO:teuthology.orchestra.run.smithi181.stdout:42949672965 2024-09-21T21:29:39.427 INFO:teuthology.orchestra.run.smithi181.stdout:34359738375 2024-09-21T21:29:39.433 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.2 2024-09-21T21:29:39.433 DEBUG:teuthology.parallel:result is None 2024-09-21T21:29:39.441 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738375 got 34359738375 for osd.1 2024-09-21T21:29:39.441 DEBUG:teuthology.parallel:result is None 2024-09-21T21:29:39.441 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T21:29:39.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:29:39.589 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.584+0000 7fe913b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.591 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.590+0000 7fe913b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.774 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:39.775 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-21T21:29:39.787 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-21T21:29:37.866054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.904601"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.990650+0000","last_change":"2024-09-21T21:29:35.997623+0000","last_active":"2024-09-21T21:29:35.990650+0000","last_peered":"2024-09-21T21:29:35.990650+0000","last_clean":"2024-09-21T21:29:35.990650+0000","last_became_active":"2024-09-21T21:29:34.126573+0000","last_became_peered":"2024-09-21T21:29:34.126573+0000","last_unstale":"2024-09-21T21:29:35.990650+0000","last_undegraded":"2024-09-21T21:29:35.990650+0000","last_fullsized":"2024-09-21T21:29:35.990650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.987779+0000","last_change":"2024-09-21T21:29:35.992900+0000","last_active":"2024-09-21T21:29:35.987779+0000","last_peered":"2024-09-21T21:29:35.987779+0000","last_clean":"2024-09-21T21:29:35.987779+0000","last_became_active":"2024-09-21T21:29:34.136145+0000","last_became_peered":"2024-09-21T21:29:34.136145+0000","last_unstale":"2024-09-21T21:29:35.987779+0000","last_undegraded":"2024-09-21T21:29:35.987779+0000","last_fullsized":"2024-09-21T21:29:35.987779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.075255+0000","last_change":"2024-09-21T21:29:36.093724+0000","last_active":"2024-09-21T21:29:36.075255+0000","last_peered":"2024-09-21T21:29:36.075255+0000","last_clean":"2024-09-21T21:29:36.075255+0000","last_became_active":"2024-09-21T21:29:34.175213+0000","last_became_peered":"2024-09-21T21:29:34.175213+0000","last_unstale":"2024-09-21T21:29:36.075255+0000","last_undegraded":"2024-09-21T21:29:36.075255+0000","last_fullsized":"2024-09-21T21:29:36.075255+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.992171+0000","last_change":"2024-09-21T21:29:35.995558+0000","last_active":"2024-09-21T21:29:35.992171+0000","last_peered":"2024-09-21T21:29:35.992171+0000","last_clean":"2024-09-21T21:29:35.992171+0000","last_became_active":"2024-09-21T21:29:34.149575+0000","last_became_peered":"2024-09-21T21:29:34.149575+0000","last_unstale":"2024-09-21T21:29:35.992171+0000","last_undegraded":"2024-09-21T21:29:35.992171+0000","last_fullsized":"2024-09-21T21:29:35.992171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'52","reported_seq":78,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.811109+0000","last_change":"2024-09-21T21:29:35.101806+0000","last_active":"2024-09-21T21:29:36.811109+0000","last_peered":"2024-09-21T21:29:36.811109+0000","last_clean":"2024-09-21T21:29:36.811109+0000","last_became_active":"2024-09-21T21:29:35.071723+0000","last_became_peered":"2024-09-21T21:29:35.071723+0000","last_unstale":"2024-09-21T21:29:36.811109+0000","last_undegraded":"2024-09-21T21:29:36.811109+0000","last_fullsized":"2024-09-21T21:29:36.811109+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:33.942786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:33.942786+0000","last_clean_scrub_stamp":"2024-09-21T21:29:33.942786+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.991177+0000","last_change":"2024-09-21T21:29:35.997327+0000","last_active":"2024-09-21T21:29:35.991177+0000","last_peered":"2024-09-21T21:29:35.991177+0000","last_clean":"2024-09-21T21:29:35.991177+0000","last_became_active":"2024-09-21T21:29:34.158755+0000","last_became_peered":"2024-09-21T21:29:34.158755+0000","last_unstale":"2024-09-21T21:29:35.991177+0000","last_undegraded":"2024-09-21T21:29:35.991177+0000","last_fullsized":"2024-09-21T21:29:35.991177+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.020884+0000","last_change":"2024-09-21T21:29:36.020884+0000","last_active":"2024-09-21T21:29:36.020884+0000","last_peered":"2024-09-21T21:29:36.020884+0000","last_clean":"2024-09-21T21:29:36.020884+0000","last_became_active":"2024-09-21T21:29:34.065896+0000","last_became_peered":"2024-09-21T21:29:34.065896+0000","last_unstale":"2024-09-21T21:29:36.020884+0000","last_undegraded":"2024-09-21T21:29:36.020884+0000","last_fullsized":"2024-09-21T21:29:36.020884+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.074577+0000","last_change":"2024-09-21T21:29:36.094075+0000","last_active":"2024-09-21T21:29:36.074577+0000","last_peered":"2024-09-21T21:29:36.074577+0000","last_clean":"2024-09-21T21:29:36.074577+0000","last_became_active":"2024-09-21T21:29:34.196978+0000","last_became_peered":"2024-09-21T21:29:34.196978+0000","last_unstale":"2024-09-21T21:29:36.074577+0000","last_undegraded":"2024-09-21T21:29:36.074577+0000","last_fullsized":"2024-09-21T21:29:36.074577+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.987947+0000","last_change":"2024-09-21T21:29:35.992996+0000","last_active":"2024-09-21T21:29:35.987947+0000","last_peered":"2024-09-21T21:29:35.987947+0000","last_clean":"2024-09-21T21:29:35.987947+0000","last_became_active":"2024-09-21T21:29:34.149640+0000","last_became_peered":"2024-09-21T21:29:34.149640+0000","last_unstale":"2024-09-21T21:29:35.987947+0000","last_undegraded":"2024-09-21T21:29:35.987947+0000","last_fullsized":"2024-09-21T21:29:35.987947+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:29:39.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:29:39.896 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.891+0000 7f6cd8d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:39.898 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:39.897+0000 7f6cd8d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:40.082 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:40.082 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-09-21T21:29:40.095 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-21T21:29:39.866487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.905034"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.990650+0000","last_change":"2024-09-21T21:29:35.997623+0000","last_active":"2024-09-21T21:29:35.990650+0000","last_peered":"2024-09-21T21:29:35.990650+0000","last_clean":"2024-09-21T21:29:35.990650+0000","last_became_active":"2024-09-21T21:29:34.126573+0000","last_became_peered":"2024-09-21T21:29:34.126573+0000","last_unstale":"2024-09-21T21:29:35.990650+0000","last_undegraded":"2024-09-21T21:29:35.990650+0000","last_fullsized":"2024-09-21T21:29:35.990650+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.987779+0000","last_change":"2024-09-21T21:29:35.992900+0000","last_active":"2024-09-21T21:29:35.987779+0000","last_peered":"2024-09-21T21:29:35.987779+0000","last_clean":"2024-09-21T21:29:35.987779+0000","last_became_active":"2024-09-21T21:29:34.136145+0000","last_became_peered":"2024-09-21T21:29:34.136145+0000","last_unstale":"2024-09-21T21:29:35.987779+0000","last_undegraded":"2024-09-21T21:29:35.987779+0000","last_fullsized":"2024-09-21T21:29:35.987779+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.075255+0000","last_change":"2024-09-21T21:29:36.093724+0000","last_active":"2024-09-21T21:29:36.075255+0000","last_peered":"2024-09-21T21:29:36.075255+0000","last_clean":"2024-09-21T21:29:36.075255+0000","last_became_active":"2024-09-21T21:29:34.175213+0000","last_became_peered":"2024-09-21T21:29:34.175213+0000","last_unstale":"2024-09-21T21:29:36.075255+0000","last_undegraded":"2024-09-21T21:29:36.075255+0000","last_fullsized":"2024-09-21T21:29:36.075255+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.992171+0000","last_change":"2024-09-21T21:29:35.995558+0000","last_active":"2024-09-21T21:29:35.992171+0000","last_peered":"2024-09-21T21:29:35.992171+0000","last_clean":"2024-09-21T21:29:35.992171+0000","last_became_active":"2024-09-21T21:29:34.149575+0000","last_became_peered":"2024-09-21T21:29:34.149575+0000","last_unstale":"2024-09-21T21:29:35.992171+0000","last_undegraded":"2024-09-21T21:29:35.992171+0000","last_fullsized":"2024-09-21T21:29:35.992171+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'52","reported_seq":78,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.811109+0000","last_change":"2024-09-21T21:29:35.101806+0000","last_active":"2024-09-21T21:29:36.811109+0000","last_peered":"2024-09-21T21:29:36.811109+0000","last_clean":"2024-09-21T21:29:36.811109+0000","last_became_active":"2024-09-21T21:29:35.071723+0000","last_became_peered":"2024-09-21T21:29:35.071723+0000","last_unstale":"2024-09-21T21:29:36.811109+0000","last_undegraded":"2024-09-21T21:29:36.811109+0000","last_fullsized":"2024-09-21T21:29:36.811109+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:33.942786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:33.942786+0000","last_clean_scrub_stamp":"2024-09-21T21:29:33.942786+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.991177+0000","last_change":"2024-09-21T21:29:35.997327+0000","last_active":"2024-09-21T21:29:35.991177+0000","last_peered":"2024-09-21T21:29:35.991177+0000","last_clean":"2024-09-21T21:29:35.991177+0000","last_became_active":"2024-09-21T21:29:34.158755+0000","last_became_peered":"2024-09-21T21:29:34.158755+0000","last_unstale":"2024-09-21T21:29:35.991177+0000","last_undegraded":"2024-09-21T21:29:35.991177+0000","last_fullsized":"2024-09-21T21:29:35.991177+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.020884+0000","last_change":"2024-09-21T21:29:36.020884+0000","last_active":"2024-09-21T21:29:36.020884+0000","last_peered":"2024-09-21T21:29:36.020884+0000","last_clean":"2024-09-21T21:29:36.020884+0000","last_became_active":"2024-09-21T21:29:34.065896+0000","last_became_peered":"2024-09-21T21:29:34.065896+0000","last_unstale":"2024-09-21T21:29:36.020884+0000","last_undegraded":"2024-09-21T21:29:36.020884+0000","last_fullsized":"2024-09-21T21:29:36.020884+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:36.074577+0000","last_change":"2024-09-21T21:29:36.094075+0000","last_active":"2024-09-21T21:29:36.074577+0000","last_peered":"2024-09-21T21:29:36.074577+0000","last_clean":"2024-09-21T21:29:36.074577+0000","last_became_active":"2024-09-21T21:29:34.196978+0000","last_became_peered":"2024-09-21T21:29:34.196978+0000","last_unstale":"2024-09-21T21:29:36.074577+0000","last_undegraded":"2024-09-21T21:29:36.074577+0000","last_fullsized":"2024-09-21T21:29:36.074577+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:29:35.987947+0000","last_change":"2024-09-21T21:29:35.992996+0000","last_active":"2024-09-21T21:29:35.987947+0000","last_peered":"2024-09-21T21:29:35.987947+0000","last_clean":"2024-09-21T21:29:35.987947+0000","last_became_active":"2024-09-21T21:29:34.149640+0000","last_became_peered":"2024-09-21T21:29:34.149640+0000","last_unstale":"2024-09-21T21:29:35.987947+0000","last_undegraded":"2024-09-21T21:29:35.987947+0000","last_fullsized":"2024-09-21T21:29:35.987947+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:29:32.937221+0000","last_clean_scrub_stamp":"2024-09-21T21:29:32.937221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:29:40.096 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T21:29:40.096 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T21:29:40.096 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T21:29:40.096 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T21:29:40.204 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:40.200+0000 7f7e238fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:40.206 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:29:40.204+0000 7f7e238fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:29:40.429 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:40.429 INFO:teuthology.orchestra.run.smithi181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T21:29:40.444 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T21:29:40.444 INFO:teuthology.run_tasks:Running task cbt... 2024-09-21T21:29:40.458 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi181.front.sepia.ceph.com', 'osds': ['ubuntu@smithi181.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi181.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi181.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi181.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-21T21:29:40.458 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-21T21:29:40.476 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:29:40.476 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:29:40.530 DEBUG:teuthology.orchestra.run.smithi181:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-21T21:29:40.586 INFO:teuthology.orchestra.run.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-21T21:29:40.891 INFO:tasks.cbt:Installing collectl 2024-09-21T21:29:40.898 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-21T21:29:40.985 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-09-21 21:29:40-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-21T21:29:41.140 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-09-21T21:29:41.153 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-09-21T21:29:41.546 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:29:41.546 INFO:teuthology.orchestra.run.smithi181.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zrFqkwx3672Cgxs82ASqcwOegUk4NMLPX_hIMNd9jmPKjV9hLsqQcwIJxE1j7Xmrsksd2Hs4fvz4Ws4sBxBueVX5w%3D%3D&use_mirror=versaweb&r= [following] 2024-09-21T21:29:41.546 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-09-21 21:29:41-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zrFqkwx3672Cgxs82ASqcwOegUk4NMLPX_hIMNd9jmPKjV9hLsqQcwIJxE1j7Xmrsksd2Hs4fvz4Ws4sBxBueVX5w%3D%3D&use_mirror=versaweb&r= 2024-09-21T21:29:41.556 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-21T21:29:41.630 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-21T21:29:41.913 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:29:41.913 INFO:teuthology.orchestra.run.smithi181.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-21T21:29:41.913 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-09-21 21:29:41-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-21T21:29:41.923 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2024-09-21T21:29:41.994 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2024-09-21T21:29:42.378 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-21T21:29:42.378 INFO:teuthology.orchestra.run.smithi181.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-21T21:29:42.378 INFO:teuthology.orchestra.run.smithi181.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-21T21:29:42.379 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:42.453 INFO:teuthology.orchestra.run.smithi181.stderr: 0K .......... .......... .......... .......... .......... 8% 665K 1s 2024-09-21T21:29:42.524 INFO:teuthology.orchestra.run.smithi181.stderr: 50K .......... .......... .......... .......... .......... 16% 705K 1s 2024-09-21T21:29:42.548 INFO:teuthology.orchestra.run.smithi181.stderr: 100K .......... .......... .......... .......... .......... 24% 2.04M 1s 2024-09-21T21:29:42.592 INFO:teuthology.orchestra.run.smithi181.stderr: 150K .......... .......... .......... .......... .......... 32% 1.12M 0s 2024-09-21T21:29:42.596 INFO:teuthology.orchestra.run.smithi181.stderr: 200K .......... .......... .......... .......... .......... 41% 10.3M 0s 2024-09-21T21:29:42.610 INFO:teuthology.orchestra.run.smithi181.stderr: 250K .......... .......... .......... .......... .......... 49% 3.51M 0s 2024-09-21T21:29:42.624 INFO:teuthology.orchestra.run.smithi181.stderr: 300K .......... .......... .......... .......... .......... 57% 3.42M 0s 2024-09-21T21:29:42.662 INFO:teuthology.orchestra.run.smithi181.stderr: 350K .......... .......... .......... .......... .......... 65% 1.32M 0s 2024-09-21T21:29:42.665 INFO:teuthology.orchestra.run.smithi181.stderr: 400K .......... .......... .......... .......... .......... 74% 14.6M 0s 2024-09-21T21:29:42.669 INFO:teuthology.orchestra.run.smithi181.stderr: 450K .......... .......... .......... .......... .......... 82% 13.0M 0s 2024-09-21T21:29:42.674 INFO:teuthology.orchestra.run.smithi181.stderr: 500K .......... .......... .......... .......... .......... 90% 9.88M 0s 2024-09-21T21:29:42.681 INFO:teuthology.orchestra.run.smithi181.stderr: 550K .......... .......... .......... .......... .......... 98% 6.81M 0s 2024-09-21T21:29:42.681 INFO:teuthology.orchestra.run.smithi181.stderr: 600K ...... 100% 11.4T=0.3s 2024-09-21T21:29:42.681 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:42.681 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21 21:29:42 (1.96 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-21T21:29:42.681 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/ 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/collectl.conf 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-21T21:29:42.714 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Socket.html 2024-09-21T21:29:42.715 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/colmux.html 2024-09-21T21:29:42.715 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-21T21:29:42.715 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-21T21:29:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Import.html 2024-09-21T21:29:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/style.css 2024-09-21T21:29:42.720 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Process.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-21T21:29:42.721 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Export.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Startup.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Performance.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Messages.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Logging.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Misc.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Examples.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-21T21:29:42.722 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Playback.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Hello.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Network.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Memory.html 2024-09-21T21:29:42.723 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Disks.html 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/service/ 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/service/collectl.service 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/ 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/misc.ph 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/gexpr.ph 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/statsd.ph 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/envrules.std 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/hello.ph 2024-09-21T21:29:42.724 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vmsum.ph 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/proctree.ph 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/README-WINDOWS 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/formatit.ph 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/colmux 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/INSTALL 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/graphite.ph 2024-09-21T21:29:42.725 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-21T21:29:42.726 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vmstat.ph 2024-09-21T21:29:42.726 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/README 2024-09-21T21:29:42.726 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/ 2024-09-21T21:29:42.726 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/collectl.1 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/colmux.1 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/lexpr.ph 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/client.pl 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/collectl 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/COPYING 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vnet.ph 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/GPL 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/UNINSTALL 2024-09-21T21:29:42.727 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/ARTISTIC 2024-09-21T21:29:42.781 INFO:teuthology.orchestra.run.smithi181.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-21T21:29:42.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-21T21:29:43.387 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:01 ago on Sat 21 Sep 2024 09:26:42 PM UTC. 2024-09-21T21:29:43.479 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-21T21:29:43.482 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-21T21:29:43.530 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-09-21T21:29:43.530 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:43.530 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout:Install 5 Packages 2024-09-21T21:29:43.531 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:43.532 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.4 M 2024-09-21T21:29:43.532 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 11 M 2024-09-21T21:29:43.532 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-09-21T21:29:43.901 INFO:teuthology.orchestra.run.smithi181.stdout:(1/5): libradospp-devel-19.3.0-5092.g62da54b4.e 204 kB/s | 31 kB 00:00 2024-09-21T21:29:43.926 INFO:teuthology.orchestra.run.smithi181.stdout:(2/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 135 kB/s | 24 kB 00:00 2024-09-21T21:29:44.002 INFO:teuthology.orchestra.run.smithi181.stdout:(3/5): perf-5.14.0-508.el9.x86_64.rpm 8.9 MB/s | 2.2 MB 00:00 2024-09-21T21:29:44.043 INFO:teuthology.orchestra.run.smithi181.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 836 kB/s | 118 kB 00:00 2024-09-21T21:29:44.069 INFO:teuthology.orchestra.run.smithi181.stdout:(5/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 92 kB/s | 13 kB 00:00 2024-09-21T21:29:44.071 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:44.072 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.4 MB/s | 2.4 MB 00:00 2024-09-21T21:29:44.102 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-09-21T21:29:44.122 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-09-21T21:29:44.123 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-09-21T21:29:44.254 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-09-21T21:29:44.255 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-09-21T21:29:44.599 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-09-21T21:29:44.679 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-21T21:29:44.709 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-21T21:29:44.749 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-21T21:29:45.076 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-21T21:29:45.154 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:29:46.874 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:29:46.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-21T21:29:46.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-21T21:29:46.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-21T21:29:46.874 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-2.34-7.el9.x86_64 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: perf-5.14.0-508.el9.x86_64 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-09-21T21:29:47.236 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-09-21T21:29:47.615 INFO:tasks.cbt:benchmark: radosbench 2024-09-21T21:29:47.616 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-21T21:29:47.641 DEBUG:teuthology.orchestra.run.smithi181:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - Settings.cluster: 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'clients': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'head': 'ubuntu@smithi181.front.sepia.ceph.com', 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'is_teuthology': True, 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'iterations': 1, 2024-09-21T21:29:48.082 INFO:teuthology.orchestra.run.smithi181.stderr: 'mons': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'osds': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'osds_per_node': 3, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'pid_dir': '/var/run/ceph', 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'pgp_size': 256, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'replication': 'replicated'}}, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'rebuild_every_test': False, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'rgws': {'ubuntu@smithi181.front.sepia.ceph.com': None}, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'use_existing': True, 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr: 'user': 'ubuntu'} 2024-09-21T21:29:48.083 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2024-09-21T21:29:48.102 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-21T21:29:48.103 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2024-09-21T21:29:48.106 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:29:48.108 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2024-09-21T21:29:48.111 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:29:48.111 INFO:teuthology.orchestra.run.smithi181.stderr:21:29:48 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:30:48.142 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:30:48.177 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:30:48.215 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-09-21T21:30:48.217 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-09-21T21:30:48.219 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2024-09-21T21:30:48.234 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/$(basename ${i}).smithi181.front.sepia.ceph.com ; done 2024-09-21T21:30:48.238 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-09-21T21:30:48.265 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - WARNING - cbt - sudo killall -9 rados 2024-09-21T21:30:48.266 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - WARNING - cbt - error 1 seen, continuing anyway... 2024-09-21T21:30:48.266 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2024-09-21T21:30:48.284 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2024-09-21T21:30:48.285 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-21T21:30:48.285 INFO:teuthology.orchestra.run.smithi181.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3 2024-09-21T21:30:48.285 INFO:teuthology.orchestra.run.smithi181.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2024-09-21T21:30:48.285 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-21T21:30:48.285 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-21T21:30:48.298 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2024-09-21T21:30:48.301 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:30:48.302 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2024-09-21T21:30:48.306 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:30:48.306 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:30:48.306 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-09-21T21:30:48.617 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-09-21T21:30:49.167 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-09-21T21:30:50.172 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:50 - INFO - cbt - Checking Health after pool creation. 2024-09-21T21:30:55.178 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:30:55.613 INFO:teuthology.orchestra.run.smithi181.stderr:21:30:55 - INFO - cbt - Final Pool Health Check. 2024-09-21T21:31:00.618 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:31:00.949 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:31:00.973 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:31:01.008 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-09-21T21:31:01.010 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-09-21T21:31:01.011 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-21T21:31:01.443 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-21T21:31:02.075 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-21T21:31:02.457 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2024-09-21T21:31:02.517 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2024-09-21T21:31:03.368 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2024-09-21T21:31:03.370 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:31:03.372 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2024-09-21T21:31:03.375 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:31:03.375 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:31:03.375 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - INFO - cbt - Running radosbench write test. 2024-09-21T21:31:03.375 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2024-09-21T21:31:03.376 INFO:teuthology.orchestra.run.smithi181.stderr:21:31:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2024-09-21T21:32:04.551 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:32:04.601 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:32:04.636 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2024-09-21T21:32:04.637 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2024-09-21T21:32:04.639 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2024-09-21T21:32:05.300 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2024-09-21T21:32:05.314 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/$(basename ${i}).smithi181.front.sepia.ceph.com ; done 2024-09-21T21:32:05.359 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - INFO - cbt - Convert results to json format. 2024-09-21T21:32:05.359 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-21T21:32:05.498 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-21T21:32:05.757 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-21T21:32:06.126 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand 2024-09-21T21:32:06.189 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2024-09-21T21:32:07.091 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl 2024-09-21T21:32:07.095 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:32:07.097 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf 2024-09-21T21:32:07.100 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:32:07.100 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:32:07.100 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - INFO - cbt - Running radosbench rand test. 2024-09-21T21:32:07.100 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.0 2024-09-21T21:32:07.101 INFO:teuthology.orchestra.run.smithi181.stderr:21:32:07 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.1 2024-09-21T21:33:08.235 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:33:08.267 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:33:08.299 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2024-09-21T21:33:08.301 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2024-09-21T21:33:08.303 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2024-09-21T21:33:08.626 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* 2024-09-21T21:33:08.639 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/$(basename ${i}).smithi181.front.sepia.ceph.com ; done 2024-09-21T21:33:08.681 INFO:teuthology.orchestra.run.smithi181.stderr:21:33:08 - INFO - cbt - Convert results to json format. 2024-09-21T21:33:08.713 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-09-21T21:33:08.721 DEBUG:teuthology.orchestra.run.smithi181:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-09-21T21:33:08.909 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-09-21T21:33:08.910 INFO:tasks.cbt_performance:reading cbt preferences 2024-09-21T21:33:08.910 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-09-21T21:33:08.974 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi181.front.sepia.ceph.com 2024-09-21T21:33:08.974 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi181.front.sepia.ceph.com 2024-09-21T21:33:08.975 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi181.front.sepia.ceph.com 2024-09-21T21:33:08.975 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi181.front.sepia.ceph.com 2024-09-21T21:33:08.976 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:33:08.976 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:33:08.995 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:33:08.995 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2024-09-21T21:33:09.051 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:33:09.052 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:33:09.053 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:33:09.053 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:33:09.108 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:33:09.108 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2024-09-21T21:33:09.164 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:33:09.164 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:33:09.164 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:33:09.165 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:33:09.225 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:33:09.225 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2024-09-21T21:33:09.282 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:33:09.282 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:33:09.282 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-09-21T21:33:09.282 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:33:09.339 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:33:09.339 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2024-09-21T21:33:09.394 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:33:09.394 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:33:09.402 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:33:09.408 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:33:09.414 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:33:09.420 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:33:09.420 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T21:33:09.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T21:33:09.560 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:33:09.551+0000 7ff486991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:09.561 INFO:teuthology.orchestra.run.smithi181.stderr:2024-09-21T21:33:09.560+0000 7ff486991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:33:09.778 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T21:33:09.778 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T21:33:09.779 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T21:33:09.779 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:33:11.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-21T21:33:15.880 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T21:33:15.880 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T21:33:15.880 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:33:15.977 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T21:33:15.978 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T21:33:15.978 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:33:16.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-21T21:33:16.294 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-21T21:33:21.496 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-21T21:33:22.079 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T21:33:22.079 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T21:33:22.079 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T21:33:22.080 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:33:22.115 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T21:33:22.115 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T21:33:22.115 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T21:33:22.115 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:33:22.168 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T21:33:22.168 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T21:33:22.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T21:33:22.253 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:33:22.253 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T21:33:22.525 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:33:22.525 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T21:33:22.635 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:33:22.635 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T21:33:22.692 INFO:tasks.ceph:Archiving mon data... 2024-09-21T21:33:22.693 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348/data/mon.a.tgz 2024-09-21T21:33:22.694 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-09-21T21:33:22.721 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.N6Mop5RS9A 2024-09-21T21:33:22.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.N6Mop5RS9A 2024-09-21T21:33:22.849 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.N6Mop5RS9A 2024-09-21T21:33:22.944 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.N6Mop5RS9A is 215KB 2024-09-21T21:33:23.037 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.N6Mop5RS9A 2024-09-21T21:33:23.052 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T21:33:23.052 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T21:33:23.176 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T21:33:23.176 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T21:33:23.178 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348/remote/smithi181/crash 2024-09-21T21:33:23.178 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T21:33:23.240 INFO:tasks.ceph:Compressing logs... 2024-09-21T21:33:23.241 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T21:33:23.314 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T21:33:23.314 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29777.log 2024-09-21T21:33:23.315 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29693.log 2024-09-21T21:33:23.315 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29353.log 2024-09-21T21:33:23.316 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29139.log 2024-09-21T21:33:23.316 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-09-21T21:33:23.316 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T21:33:23.316 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29273.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30699.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30672.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28994.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29355.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30967.log 2024-09-21T21:33:23.317 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29050.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29607.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28459.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29189.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31267.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29480.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30966.log 2024-09-21T21:33:23.319 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28660.log 2024-09-21T21:33:23.320 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29567.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30745.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28848.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29735.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28371.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29481.log 2024-09-21T21:33:23.321 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28818.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29441.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27413.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29354.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28609.log 2024-09-21T21:33:23.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28569.log 2024-09-21T21:33:23.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T21:33:23.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30608.log 2024-09-21T21:33:23.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29606.log 2024-09-21T21:33:23.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29231.log 2024-09-21T21:33:23.323 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T21:33:23.324 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30722.log 2024-09-21T21:33:23.324 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30809.log 2024-09-21T21:33:23.324 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29075.log 2024-09-21T21:33:23.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29353.log: /var/log/ceph/ceph-client.admin.29693.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.29480.log: /var/log/ceph/ceph-client.admin.30967.log: /var/log/ceph/ceph-client.admin.29567.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.31267.log: /var/log/ceph/ceph-client.admin.30966.log: 66.4%/var/log/ceph/ceph-mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.30967.log.gz 2024-09-21T21:33:23.343 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31267.log.gz 2024-09-21T21:33:23.343 INFO:teuthology.orchestra.run.smithi181.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30966.log.gz 2024-09-21T21:33:23.344 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.29777.log: /var/log/ceph/ceph-client.admin.29273.log: /var/log/ceph/ceph-client.admin.30699.log: /var/log/ceph/ceph-client.admin.29139.log: /var/log/ceph/ceph-client.admin.30672.log: /var/log/ceph/ceph-client.admin.30810.log: /var/log/ceph/ceph-client.admin.30608.log: /var/log/ceph/ceph-client.admin.28371.log: /var/log/ceph/ceph-client.admin.28660.log: /var/log/ceph/ceph-client.admin.28994.log: /var/log/ceph/ceph-client.admin.29355.log: /var/log/ceph/ceph-client.admin.29050.log: /var/log/ceph/ceph-client.admin.29231.log: /var/log/ceph/ceph-client.admin.28818.log: /var/log/ceph/ceph-client.admin.29607.log: /var/log/ceph/ceph-client.admin.29189.log: /var/log/ceph/ceph-client.admin.28459.log: /var/log/ceph/ceph-client.admin.30745.log: /var/log/ceph/ceph-client.admin.29354.log: /var/log/ceph/ceph-client.admin.28848.log: /var/log/ceph/ceph-client.admin.29481.log: /var/log/ceph/ceph.tmp-client.admin.27413.log: /var/log/ceph/ceph-client.admin.29735.log: /var/log/ceph/ceph-client.admin.28569.log: /var/log/ceph/ceph-client.admin.29441.log: /var/log/ceph/ceph-client.admin.28609.log: /var/log/ceph/ceph-client.admin.29606.log: /var/log/ceph/ceph-client.admin.30809.log: /var/log/ceph/ceph-client.admin.30722.log: /var/log/ceph/ceph-client.admin.29075.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30672.log.gz 2024-09-21T21:33:23.346 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30608.log.gz 2024-09-21T21:33:23.347 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29353.log.gz 2024-09-21T21:33:23.348 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29355.log.gz 2024-09-21T21:33:23.348 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29480.log.gz 2024-09-21T21:33:23.349 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29481.log.gz 2024-09-21T21:33:23.349 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29354.log.gz 2024-09-21T21:33:23.349 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29441.log.gz 2024-09-21T21:33:23.349 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29567.log.gz 2024-09-21T21:33:23.349 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29607.log.gz 2024-09-21T21:33:23.350 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29606.log.gz 2024-09-21T21:33:23.350 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29693.log.gz 2024-09-21T21:33:23.350 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29735.log.gz 2024-09-21T21:33:23.350 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29777.log.gz 2024-09-21T21:33:23.362 INFO:teuthology.orchestra.run.smithi181.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T21:33:23.364 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30699.log.gz 2024-09-21T21:33:23.364 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30722.log.gz 2024-09-21T21:33:23.364 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30745.log.gz 2024-09-21T21:33:23.368 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28569.log.gz 2024-09-21T21:33:23.369 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28609.log.gz 2024-09-21T21:33:23.369 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28660.log.gz 2024-09-21T21:33:23.371 INFO:teuthology.orchestra.run.smithi181.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-09-21T21:33:23.371 INFO:teuthology.orchestra.run.smithi181.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.30809.log.gz 2024-09-21T21:33:23.373 INFO:teuthology.orchestra.run.smithi181.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28818.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28994.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29050.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.29075.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29139.log.gz 2024-09-21T21:33:23.385 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29189.log.gz 2024-09-21T21:33:23.386 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29231.log.gz 2024-09-21T21:33:23.386 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29273.log.gz 2024-09-21T21:33:23.449 INFO:teuthology.orchestra.run.smithi181.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T21:33:23.563 INFO:teuthology.orchestra.run.smithi181.stderr: 66.4% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27413.log.gz 2024-09-21T21:33:23.564 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28371.log.gz 2024-09-21T21:33:23.564 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28459.log.gz 2024-09-21T21:33:23.564 INFO:teuthology.orchestra.run.smithi181.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T21:33:26.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-21T21:33:31.217 INFO:teuthology.orchestra.run.smithi181.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T21:33:32.966 INFO:teuthology.orchestra.run.smithi181.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T21:33:33.005 INFO:teuthology.orchestra.run.smithi181.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T21:33:33.007 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-09-21T21:33:33.007 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m9.712s 2024-09-21T21:33:33.007 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m4.959s 2024-09-21T21:33:33.007 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.255s 2024-09-21T21:33:33.007 INFO:tasks.ceph:Archiving logs... 2024-09-21T21:33:33.008 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348/remote/smithi181/log 2024-09-21T21:33:33.009 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T21:33:34.073 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T21:33:34.082 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T21:33:34.082 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T21:33:34.138 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T21:33:34.146 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T21:33:34.146 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:33:34.167 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:33:34.232 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:33:34.232 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-09-21T21:33:34.232 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv01.front.sepia.ceph.com 3 6 377 34 +235us[ +183us] +/- 39ms 2024-09-21T21:33:34.232 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -828us[ -828us] +/- 56ms 2024-09-21T21:33:34.232 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 +518us[ +518us] +/- 72ms 2024-09-21T21:33:34.233 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:33:34.238 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T21:33:34.247 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T21:33:34.247 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T21:33:34.272 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T21:33:34.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:33:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:33:34.398 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T21:33:34.407 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T21:33:34.439 INFO:teuthology.task.internal:Duration was 651.432301 seconds 2024-09-21T21:33:34.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T21:33:34.512 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T21:33:34.512 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T21:33:34.680 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:33:34.985 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T21:33:34.986 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-09-21T21:33:34.986 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T21:33:35.006 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T21:33:35.006 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T21:33:35.073 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T21:33:35.073 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T21:33:35.223 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T21:33:35.233 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T21:33:35.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T21:33:35.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T21:33:35.272 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:33:35.346 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-09-21T21:33:35.367 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:33:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:33:35.423 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T21:33:35.433 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T21:33:35.434 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914348/remote/smithi181 2024-09-21T21:33:35.434 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T21:33:35.611 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T21:33:35.611 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T21:33:35.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T21:33:35.679 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T21:33:35.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T21:33:35.723 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T21:33:35.723 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T21:33:35.738 INFO:teuthology.orchestra.run.smithi181.stdout: 265336 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 21:33 /home/ubuntu/cephtest 2024-09-21T21:33:35.739 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T21:33:35.756 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T21:33:35.842 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7914348 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4K_rand_read} duration: 651.4323010444641 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-21T21:33:35.842 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:33:35.931 INFO:teuthology.run:pass