2024-09-21T21:21:59.740 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T21:21:59.760 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:21:59.850 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_rand_read} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914352' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwD5kKDz+RybxrtKyJnzvWJkKwZzaktx4IwSY0JNT1hEzZMe/q4CDC+18ssmfeNfvyPVrOhS6fh9BBH1VazQIQ= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T21:21:59.851 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T21:21:59.851 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T21:21:59.852 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T21:21:59.853 INFO:teuthology.task.internal:Checking packages... 2024-09-21T21:21:59.873 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T21:21:59.873 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T21:21:59.873 INFO:teuthology.packaging:ref: None 2024-09-21T21:21:59.873 INFO:teuthology.packaging:tag: None 2024-09-21T21:21:59.873 INFO:teuthology.packaging:branch: main 2024-09-21T21:21:59.873 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:21:59.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T21:22:00.070 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T21:22:00.071 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T21:22:00.076 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T21:22:00.076 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T21:22:00.081 INFO:teuthology.task.internal:Saving configuration 2024-09-21T21:22:00.097 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T21:22:00.104 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T21:22:00.124 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:16:54.431834', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwD5kKDz+RybxrtKyJnzvWJkKwZzaktx4IwSY0JNT1hEzZMe/q4CDC+18ssmfeNfvyPVrOhS6fh9BBH1VazQIQ='} 2024-09-21T21:22:00.124 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T21:22:00.130 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-21T21:22:00.130 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T21:22:00.166 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f72b42ec160>, signals=[15]) 2024-09-21T21:22:00.167 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T21:22:00.173 INFO:teuthology.task.internal:Opening connections... 2024-09-21T21:22:00.173 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:22:00.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:22:00.249 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T21:22:00.257 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-09-21T21:22:00.336 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-09-21T21:22:00.336 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-09-21T21:22:00.393 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T21:22:00.394 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T21:22:00.395 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-09-21T21:22:00.423 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T21:22:00.441 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T21:22:00.561 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T21:22:00.562 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:22:00.579 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T21:22:00.585 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T21:22:00.586 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:22:00.643 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T21:22:00.676 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T21:22:00.689 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T21:22:00.689 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T21:22:00.690 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:22:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:22:00.707 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-21T21:22:00.764 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-503.el9.x86_64 2024-09-21T21:22:00.765 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-503.el9.x86_64 2024-09-21T21:22:00.765 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-09-21T21:22:03.105 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-21T21:22:05.198 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-21T21:22:10.773 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-21T21:22:12.282 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-21T21:22:13.462 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-21T21:22:19.284 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T21:22:21.274 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T21:22:21.274 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T21:22:21.371 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:22:21.386 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-09-21T21:22:21.387 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:22:21.388 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 79 M 2024-09-21T21:22:21.388 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 126 M 2024-09-21T21:22:21.388 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:22:22.069 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 63 kB/s | 36 kB 00:00 2024-09-21T21:22:23.027 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-21T21:22:23.586 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-21T21:22:24.469 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 12 MB/s | 29 MB 00:02 2024-09-21T21:22:24.469 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:22:24.470 INFO:teuthology.orchestra.run.smithi063.stdout:Total 26 MB/s | 79 MB 00:03 2024-09-21T21:22:25.182 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:22:25.568 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:22:26.538 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:22:26.541 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:22:29.089 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:22:29.773 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:22:29.863 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:22:32.605 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:22:32.732 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:22:38.020 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:22:38.066 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:22:43.209 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:22.322 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:22.363 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:23:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:23:22.641 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.457 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:23:23.458 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:23:23.796 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:23:23.873 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.874 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:23:23.874 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T21:23:23.874 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:23:23.874 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:23:23.874 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T21:23:23.874 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:23:23.874 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:23:23.932 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:23:23.933 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:23:23.933 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 82.el9 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-09-21T21:23:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8274670 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T21:23:23.972 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T21:23:23.973 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T21:23:23.973 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-09-21T21:23:23.973 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:23:23.973 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T21:23:23.974 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T21:23:23.974 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-508.el9.x86_64 2024-09-21T21:23:23.974 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T21:23:24.589 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-09-21T21:23:25.859 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T21:23:25.888 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-09-21T21:23:25.890 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-09-21T21:23:25.904 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.0fbXl4FThT 2024-09-21T21:23:25.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0fbXl4FThT 2024-09-21T21:23:25.970 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.0fbXl4FThT 2024-09-21T21:23:26.107 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.0fbXl4FThT is 6KB 2024-09-21T21:23:26.155 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.0fbXl4FThT 2024-09-21T21:23:26.170 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T21:23:26.238 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T21:23:26.238 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T21:23:26.238 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T21:23:26.238 INFO:teuthology.orchestra.run.smithi063.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T21:23:26.240 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T21:23:26.385 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-09-21T21:23:56.414 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T21:23:56.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:23:56.416 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:14.959 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-21T21:24:23.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:24:23.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:27.027 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-21T21:24:39.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:24:39.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:24:57.455 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-09-21T21:25:12.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:25:12.465 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:25:12.779 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-21T21:25:13.133 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:25:13.133 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T21:25:13.133 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-09-21T21:25:13.149 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-508.el9.x86_64 2024-09-21T21:25:13.149 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T21:25:13.149 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:25:13.149 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T21:25:14.151 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T21:25:14.151 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:25:14.276 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-09-21T21:25:14.309 DEBUG:teuthology.parallel:result is None 2024-09-21T21:25:14.309 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T21:25:14.316 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T21:25:14.317 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:25:14.332 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T21:25:14.340 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T21:25:14.354 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T21:25:14.354 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:25:14.540 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T21:25:14.555 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T21:25:14.556 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:25:14.582 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:25:14.593 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:25:14.594 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T21:25:14.611 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T21:25:14.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:25:14.666 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T21:25:14.676 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T21:25:14.676 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:25:14.728 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:25:14.812 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:25:14.884 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:25:14.972 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:25:15.016 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:25:15.016 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:25:15.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-09-21T21:25:15.172 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:25:15.462 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T21:25:15.477 INFO:teuthology.task.internal:Starting timer... 2024-09-21T21:25:15.477 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T21:25:15.488 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T21:25:15.517 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-09-21T21:25:15.586 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-09-21T21:25:15.587 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T21:25:15.587 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-09-21T21:25:15.631 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-09-21T21:25:15.632 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive'} 2024-09-21T21:25:15.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:25:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:25:15.703 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T21:25:15.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-09-21T21:25:15.842 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T21:25:15.852 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T21:25:15.863 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T21:25:15.864 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T21:29:00.638 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com')] 2024-09-21T21:29:00.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:29:00.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:29:00.721 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-09-21T21:29:00.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-09-21T21:29:00.799 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T21:29:00.815 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T21:29:00.815 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T21:29:00.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:29:00.872 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T21:29:00.888 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T21:29:00.914 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-09-21T21:29:00.925 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-09-21T21:29:00.940 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T21:29:00.955 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T21:29:01.011 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:29:01.014 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:29:01.015 INFO:teuthology.run_tasks:Running task install... 2024-09-21T21:29:01.024 DEBUG:teuthology.task.install:project ceph 2024-09-21T21:29:01.024 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T21:29:01.024 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T21:29:01.024 INFO:teuthology.task.install:Using flavor: crimson 2024-09-21T21:29:01.029 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T21:29:01.029 INFO:teuthology.task.install:extra packages: [] 2024-09-21T21:29:01.029 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T21:29:01.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:29:01.180 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-21T21:29:01.181 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-21T21:29:01.292 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T21:29:01.292 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:29:01.292 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T21:29:01.320 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T21:29:01.320 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T21:29:01.388 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T21:29:01.467 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T21:29:01.532 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-09-21T21:29:01.534 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-09-21T21:29:01.787 INFO:teuthology.orchestra.run.smithi063.stdout:45 files removed 2024-09-21T21:29:01.815 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-09-21T21:29:02.377 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 336 kB/s | 88 kB 00:00 2024-09-21T21:29:02.669 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-21T21:29:02.944 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 8.8 kB/s | 2.1 kB 00:00 2024-09-21T21:29:03.711 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T21:29:09.053 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 4.8 MB/s | 20 MB 00:04 2024-09-21T21:29:13.404 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-21T21:29:14.872 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 19 kB 00:00 2024-09-21T21:29:16.117 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-21T21:29:21.665 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-09-21T21:29:23.951 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:29:23.967 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:23.967 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:29:23.967 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:23.967 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T21:29:23.968 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-21T21:29:23.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:23.970 INFO:teuthology.orchestra.run.smithi063.stdout:Install 31 Packages 2024-09-21T21:29:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-09-21T21:29:23.971 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 60 M 2024-09-21T21:29:23.971 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:29:24.708 INFO:teuthology.orchestra.run.smithi063.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 9.2 MB/s | 6.1 MB 00:00 2024-09-21T21:29:24.833 INFO:teuthology.orchestra.run.smithi063.stdout:(2/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 13 MB/s | 10 MB 00:00 2024-09-21T21:29:24.858 INFO:teuthology.orchestra.run.smithi063.stdout:(3/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 168 kB/s | 25 kB 00:00 2024-09-21T21:29:24.900 INFO:teuthology.orchestra.run.smithi063.stdout:(4/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 12 MB/s | 808 kB 00:00 2024-09-21T21:29:24.934 INFO:teuthology.orchestra.run.smithi063.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 7.2 MB/s | 558 kB 00:00 2024-09-21T21:29:24.992 INFO:teuthology.orchestra.run.smithi063.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 776 kB/s | 45 kB 00:00 2024-09-21T21:29:25.084 INFO:teuthology.orchestra.run.smithi063.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 1.6 MB/s | 148 kB 00:00 2024-09-21T21:29:25.160 INFO:teuthology.orchestra.run.smithi063.stdout:(8/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_6 21 MB/s | 5.4 MB 00:00 2024-09-21T21:29:25.185 INFO:teuthology.orchestra.run.smithi063.stdout:(9/33): python3-cephfs-19.3.0-5092.g62da54b4.el 1.6 MB/s | 159 kB 00:00 2024-09-21T21:29:25.210 INFO:teuthology.orchestra.run.smithi063.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 6.1 MB/s | 314 kB 00:00 2024-09-21T21:29:25.244 INFO:teuthology.orchestra.run.smithi063.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-21T21:29:25.269 INFO:teuthology.orchestra.run.smithi063.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-21T21:29:25.469 INFO:teuthology.orchestra.run.smithi063.stdout:(13/33): ceph-common-19.3.0-5092.g62da54b4.el9. 14 MB/s | 20 MB 00:01 2024-09-21T21:29:26.078 INFO:teuthology.orchestra.run.smithi063.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 128 kB/s | 107 kB 00:00 2024-09-21T21:29:26.145 INFO:teuthology.orchestra.run.smithi063.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 195 kB/s | 171 kB 00:00 2024-09-21T21:29:26.245 INFO:teuthology.orchestra.run.smithi063.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 207 kB/s | 160 kB 00:00 2024-09-21T21:29:26.662 INFO:teuthology.orchestra.run.smithi063.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 77 kB/s | 45 kB 00:00 2024-09-21T21:29:27.087 INFO:teuthology.orchestra.run.smithi063.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 347 kB/s | 292 kB 00:00 2024-09-21T21:29:27.229 INFO:teuthology.orchestra.run.smithi063.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 74 kB/s | 42 kB 00:00 2024-09-21T21:29:27.496 INFO:teuthology.orchestra.run.smithi063.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 490 kB/s | 662 kB 00:01 2024-09-21T21:29:27.830 INFO:teuthology.orchestra.run.smithi063.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 60 kB/s | 36 kB 00:00 2024-09-21T21:29:27.855 INFO:teuthology.orchestra.run.smithi063.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 288 kB/s | 221 kB 00:00 2024-09-21T21:29:27.922 INFO:teuthology.orchestra.run.smithi063.stdout:(23/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-21T21:29:27.947 INFO:teuthology.orchestra.run.smithi063.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-21T21:29:27.980 INFO:teuthology.orchestra.run.smithi063.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-09-21T21:29:28.014 INFO:teuthology.orchestra.run.smithi063.stdout:(26/33): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-09-21T21:29:28.081 INFO:teuthology.orchestra.run.smithi063.stdout:(27/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 81 kB/s | 47 kB 00:00 2024-09-21T21:29:28.123 INFO:teuthology.orchestra.run.smithi063.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 839 kB 00:00 2024-09-21T21:29:28.165 INFO:teuthology.orchestra.run.smithi063.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 2.2 MB/s | 191 kB 00:00 2024-09-21T21:29:28.273 INFO:teuthology.orchestra.run.smithi063.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-21T21:29:28.374 INFO:teuthology.orchestra.run.smithi063.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.7 MB/s | 4.4 MB 00:00 2024-09-21T21:29:28.449 INFO:teuthology.orchestra.run.smithi063.stdout:(32/33): librados2-19.3.0-5092.g62da54b4.el9.x8 12 MB/s | 3.5 MB 00:00 2024-09-21T21:29:28.574 INFO:teuthology.orchestra.run.smithi063.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 13 MB/s | 3.8 MB 00:00 2024-09-21T21:29:28.579 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:28.580 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 60 MB 00:04 2024-09-21T21:29:28.698 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:29:28.739 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:29:28.739 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:29:29.415 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:29:29.416 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:29:30.945 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:29:31.468 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T21:29:31.568 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:31.660 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:31.761 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:29:31.786 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:31.840 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:31.894 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-21T21:29:31.932 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-21T21:29:31.957 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-21T21:29:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:29:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:29:32.218 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:29:32.255 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:29:32.290 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-21T21:29:32.334 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-21T21:29:32.363 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:29:32.462 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-21T21:29:32.484 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:29:32.581 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:29:32.617 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-21T21:29:32.729 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-21T21:29:32.766 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-21T21:29:32.806 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-21T21:29:32.849 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-21T21:29:32.887 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-21T21:29:32.923 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-21T21:29:33.298 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T21:29:33.349 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:29:33.440 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:29:33.471 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-21T21:29:33.507 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T21:29:34.117 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T21:29:34.143 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:29:34.231 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:29:34.317 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-21T21:29:34.354 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-21T21:29:36.937 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:36.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:37.732 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:29:37.770 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:29:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:29:37.870 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T21:29:37.870 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T21:29:37.870 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:37.904 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-09-21T21:29:48.493 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:49.644 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:29:49.671 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:49.673 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:29:49.763 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:29:49.763 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:51.031 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:51.031 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-21T21:29:51.032 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T21:29:51.033 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T21:29:51.034 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:29:51.035 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T21:29:51.035 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.910 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.911 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T21:29:51.912 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T21:29:51.913 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:51.913 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:29:52.239 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-09-21T21:29:52.843 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:31 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:29:52.988 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-21T21:29:52.989 INFO:teuthology.orchestra.run.smithi063.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout:Install 9 Packages 2024-09-21T21:29:52.990 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:29:52.991 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 83 M 2024-09-21T21:29:52.991 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 775 M 2024-09-21T21:29:52.991 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:29:53.442 INFO:teuthology.orchestra.run.smithi063.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 397 kB/s | 94 kB 00:00 2024-09-21T21:29:53.467 INFO:teuthology.orchestra.run.smithi063.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 667 kB/s | 175 kB 00:00 2024-09-21T21:29:53.592 INFO:teuthology.orchestra.run.smithi063.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 183 kB 00:00 2024-09-21T21:29:53.618 INFO:teuthology.orchestra.run.smithi063.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.9 MB/s | 513 kB 00:00 2024-09-21T21:29:53.693 INFO:teuthology.orchestra.run.smithi063.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-09-21T21:29:53.726 INFO:teuthology.orchestra.run.smithi063.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.7 MB/s | 304 kB 00:00 2024-09-21T21:29:53.768 INFO:teuthology.orchestra.run.smithi063.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-09-21T21:29:53.793 INFO:teuthology.orchestra.run.smithi063.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.8 MB/s | 122 kB 00:00 2024-09-21T21:29:55.544 INFO:teuthology.orchestra.run.smithi063.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 34 MB/s | 81 MB 00:02 2024-09-21T21:29:55.547 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:29:55.547 INFO:teuthology.orchestra.run.smithi063.stdout:Total 32 MB/s | 83 MB 00:02 2024-09-21T21:29:55.634 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:29:55.659 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:29:55.660 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:29:56.281 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:29:56.282 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:29:57.181 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:29:57.384 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-21T21:29:57.435 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-21T21:29:57.482 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-21T21:29:57.515 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-21T21:29:57.548 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:29:57.571 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:29:57.642 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:29:57.681 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-21T21:29:57.707 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:30:09.181 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:30:09.274 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:30:15.211 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:30:15.211 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-21T21:30:15.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-21T21:30:15.638 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T21:30:15.639 INFO:teuthology.orchestra.run.smithi063.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-21T21:30:15.640 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:15.640 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:16.078 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-09-21T21:30:16.686 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:16.839 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:16.841 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:30:16.841 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:16.841 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:30:16.841 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T21:30:16.842 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T21:30:16.843 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T21:30:16.844 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:30:16.845 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:16.846 INFO:teuthology.orchestra.run.smithi063.stdout:Install 37 Packages 2024-09-21T21:30:16.846 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:16.847 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 84 M 2024-09-21T21:30:16.847 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 896 M 2024-09-21T21:30:16.847 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:30:17.535 INFO:teuthology.orchestra.run.smithi063.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-21T21:30:17.735 INFO:teuthology.orchestra.run.smithi063.stdout:(2/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 4.8 MB/s | 1.6 MB 00:00 2024-09-21T21:30:17.793 INFO:teuthology.orchestra.run.smithi063.stdout:(3/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 5.6 MB/s | 2.2 MB 00:00 2024-09-21T21:30:17.885 INFO:teuthology.orchestra.run.smithi063.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 3.1 MB/s | 292 kB 00:00 2024-09-21T21:30:17.969 INFO:teuthology.orchestra.run.smithi063.stdout:(5/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 12 MB/s | 5.1 MB 00:00 2024-09-21T21:30:18.436 INFO:teuthology.orchestra.run.smithi063.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 460 kB/s | 253 kB 00:00 2024-09-21T21:30:18.552 INFO:teuthology.orchestra.run.smithi063.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 915 kB/s | 106 kB 00:00 2024-09-21T21:30:18.670 INFO:teuthology.orchestra.run.smithi063.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-21T21:30:18.711 INFO:teuthology.orchestra.run.smithi063.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 1.7 MB/s | 1.2 MB 00:00 2024-09-21T21:30:18.787 INFO:teuthology.orchestra.run.smithi063.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-09-21T21:30:18.829 INFO:teuthology.orchestra.run.smithi063.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-21T21:30:19.004 INFO:teuthology.orchestra.run.smithi063.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 870 kB/s | 188 kB 00:00 2024-09-21T21:30:19.054 INFO:teuthology.orchestra.run.smithi063.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 765 kB/s | 172 kB 00:00 2024-09-21T21:30:19.079 INFO:teuthology.orchestra.run.smithi063.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 463 kB/s | 35 kB 00:00 2024-09-21T21:30:19.104 INFO:teuthology.orchestra.run.smithi063.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-21T21:30:19.130 INFO:teuthology.orchestra.run.smithi063.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-21T21:30:19.155 INFO:teuthology.orchestra.run.smithi063.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 828 kB/s | 42 kB 00:00 2024-09-21T21:30:19.189 INFO:teuthology.orchestra.run.smithi063.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 381 kB/s | 22 kB 00:00 2024-09-21T21:30:19.823 INFO:teuthology.orchestra.run.smithi063.stdout:(19/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 34 MB/s | 70 MB 00:02 2024-09-21T21:30:19.873 INFO:teuthology.orchestra.run.smithi063.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 211 kB/s | 151 kB 00:00 2024-09-21T21:30:19.898 INFO:teuthology.orchestra.run.smithi063.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 61 kB/s | 43 kB 00:00 2024-09-21T21:30:19.923 INFO:teuthology.orchestra.run.smithi063.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 426 kB/s | 11 kB 00:00 2024-09-21T21:30:19.940 INFO:teuthology.orchestra.run.smithi063.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-21T21:30:19.966 INFO:teuthology.orchestra.run.smithi063.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2024-09-21T21:30:19.991 INFO:teuthology.orchestra.run.smithi063.stdout:(25/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-21T21:30:20.008 INFO:teuthology.orchestra.run.smithi063.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-09-21T21:30:20.025 INFO:teuthology.orchestra.run.smithi063.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-09-21T21:30:20.042 INFO:teuthology.orchestra.run.smithi063.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-21T21:30:20.059 INFO:teuthology.orchestra.run.smithi063.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 1.4 MB/s | 46 kB 00:00 2024-09-21T21:30:20.075 INFO:teuthology.orchestra.run.smithi063.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T21:30:20.092 INFO:teuthology.orchestra.run.smithi063.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 493 kB/s | 16 kB 00:00 2024-09-21T21:30:20.117 INFO:teuthology.orchestra.run.smithi063.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-21T21:30:20.134 INFO:teuthology.orchestra.run.smithi063.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 856 kB/s | 36 kB 00:00 2024-09-21T21:30:20.168 INFO:teuthology.orchestra.run.smithi063.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-21T21:30:20.193 INFO:teuthology.orchestra.run.smithi063.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T21:30:20.218 INFO:teuthology.orchestra.run.smithi063.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-09-21T21:30:20.244 INFO:teuthology.orchestra.run.smithi063.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T21:30:20.250 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:20.250 INFO:teuthology.orchestra.run.smithi063.stdout:Total 25 MB/s | 84 MB 00:03 2024-09-21T21:30:20.407 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:30:20.454 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:30:20.454 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:30:21.287 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:30:21.288 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:30:23.129 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:30:23.180 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T21:30:23.213 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T21:30:23.240 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T21:30:23.306 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T21:30:23.347 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T21:30:23.381 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T21:30:23.476 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T21:30:23.692 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T21:30:23.734 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T21:30:23.777 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T21:30:23.814 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T21:30:23.848 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T21:30:23.881 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T21:30:23.930 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T21:30:23.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T21:30:24.008 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T21:30:24.057 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T21:30:24.097 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T21:30:24.173 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T21:30:24.324 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T21:30:24.415 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T21:30:24.478 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T21:30:24.515 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T21:30:24.573 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T21:30:24.645 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T21:30:24.840 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T21:30:24.893 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T21:30:24.974 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T21:30:25.072 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T21:30:25.111 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T21:30:25.251 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T21:30:25.463 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-21T21:30:25.493 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:30:25.521 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:30:25.522 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:25.522 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T21:30:25.522 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:30:25.522 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:30:25.522 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:36.686 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:30:36.714 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:30:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:36.714 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T21:30:36.715 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:30:36.715 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:30:36.715 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:37.286 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:30:37.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:37.562 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:30:37.587 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:37.653 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-21T21:30:44.147 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T21:30:44.148 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T21:30:44.149 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T21:30:44.150 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T21:30:44.150 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:44.968 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T21:30:44.969 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T21:30:44.970 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:44.971 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:45.285 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-09-21T21:30:45.904 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:45.999 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:30:46.047 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:46.048 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:30:46.048 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:46.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-09-21T21:30:46.726 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:46.864 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:46.865 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:30:46.866 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:46.866 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 784 k 2024-09-21T21:30:46.866 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 791 k 2024-09-21T21:30:46.866 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:30:47.136 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.8 MB/s | 784 kB 00:00 2024-09-21T21:30:47.136 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:47.137 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.8 MB/s | 784 kB 00:00 2024-09-21T21:30:47.137 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:30:47.143 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:30:47.143 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:30:47.149 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:30:47.149 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:30:47.229 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:30:47.666 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:47.752 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:48.126 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:48.388 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:30:48.388 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:48.388 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:30:48.388 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:30:48.389 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:48.389 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:48.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T21:30:49.229 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:30:49.366 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:49.367 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 158 k 2024-09-21T21:30:49.368 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 644 k 2024-09-21T21:30:49.368 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:30:49.574 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 766 kB/s | 158 kB 00:00 2024-09-21T21:30:49.574 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:49.575 INFO:teuthology.orchestra.run.smithi063.stdout:Total 759 kB/s | 158 kB 00:00 2024-09-21T21:30:49.575 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:30:49.586 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:30:49.586 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:30:49.646 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:30:49.647 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:30:49.794 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:30:49.850 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:49.877 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:49.877 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:30:49.878 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T21:30:49.878 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:50.800 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:51.020 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-09-21T21:30:51.624 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:51.715 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:30:51.760 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:51.761 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:30:51.761 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:30:51.832 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T21:30:52.437 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:30:52.578 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T21:30:52.579 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout:Install 11 Packages 2024-09-21T21:30:52.580 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:30:52.581 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 18 M 2024-09-21T21:30:52.581 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 156 M 2024-09-21T21:30:52.581 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:30:53.044 INFO:teuthology.orchestra.run.smithi063.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 230 kB/s | 34 kB 00:00 2024-09-21T21:30:53.069 INFO:teuthology.orchestra.run.smithi063.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 95 kB/s | 16 kB 00:00 2024-09-21T21:30:53.186 INFO:teuthology.orchestra.run.smithi063.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-21T21:30:53.244 INFO:teuthology.orchestra.run.smithi063.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-09-21T21:30:53.295 INFO:teuthology.orchestra.run.smithi063.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-09-21T21:30:53.328 INFO:teuthology.orchestra.run.smithi063.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.8 MB/s | 548 kB 00:00 2024-09-21T21:30:53.362 INFO:teuthology.orchestra.run.smithi063.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 4.3 MB/s | 144 kB 00:00 2024-09-21T21:30:53.387 INFO:teuthology.orchestra.run.smithi063.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T21:30:53.421 INFO:teuthology.orchestra.run.smithi063.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-21T21:30:53.583 INFO:teuthology.orchestra.run.smithi063.stdout:(10/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 21 MB/s | 14 MB 00:00 2024-09-21T21:30:53.638 INFO:teuthology.orchestra.run.smithi063.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.9 MB/s | 2.0 MB 00:00 2024-09-21T21:30:53.642 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:30:53.642 INFO:teuthology.orchestra.run.smithi063.stdout:Total 17 MB/s | 18 MB 00:01 2024-09-21T21:30:53.673 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:30:53.693 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:30:53.693 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:30:53.964 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:30:53.964 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:30:54.562 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:30:54.655 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-21T21:30:54.683 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-21T21:30:54.787 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-21T21:30:54.925 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-21T21:30:54.978 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-21T21:30:55.067 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-21T21:30:55.114 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-21T21:30:55.141 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:30:55.165 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-21T21:30:58.610 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-21T21:30:58.661 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:31:00.616 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:31:00.616 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-21T21:31:00.616 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-21T21:31:00.616 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-21T21:31:00.617 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T21:31:00.941 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:00.942 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:01.190 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T21:31:01.795 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:01.934 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:01.935 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-21T21:31:01.935 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:31:01.935 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T21:31:01.936 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 38 M 2024-09-21T21:31:01.937 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 212 M 2024-09-21T21:31:01.938 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:02.576 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-21T21:31:02.626 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-21T21:31:02.709 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 33 kB 00:00 2024-09-21T21:31:02.760 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-21T21:31:02.810 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 773 kB/s | 38 kB 00:00 2024-09-21T21:31:02.902 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): ceph-mgr-diskprediction-local-19.3.0-50 16 MB/s | 7.4 MB 00:00 2024-09-21T21:31:03.069 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.8 MB/s | 3.0 MB 00:00 2024-09-21T21:31:03.169 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 918 kB/s | 245 kB 00:00 2024-09-21T21:31:03.236 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2024-09-21T21:31:03.261 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.7 MB/s | 250 kB 00:00 2024-09-21T21:31:03.378 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-21T21:31:03.837 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-09-21T21:31:03.837 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:03.838 INFO:teuthology.orchestra.run.smithi063.stdout:Total 20 MB/s | 38 MB 00:01 2024-09-21T21:31:04.202 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:04.230 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:04.230 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:04.681 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:04.681 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:05.471 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:05.556 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T21:31:05.594 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:31:05.699 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:31:05.737 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T21:31:05.929 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T21:31:05.960 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T21:31:06.039 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T21:31:06.735 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T21:31:06.852 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T21:31:08.176 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:31:09.264 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T21:31:09.308 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:31:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:31:11.797 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T21:31:11.798 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T21:31:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T21:31:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:31:11.799 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T21:31:12.208 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T21:31:12.208 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:12.208 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:12.208 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:12.208 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:12.209 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:12.578 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-09-21T21:31:13.190 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:52 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:13.329 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:13.330 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:13.330 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:31:13.330 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:13.330 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T21:31:13.331 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.2 M 2024-09-21T21:31:13.332 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-09-21T21:31:13.333 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:13.551 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 228 kB/s | 19 kB 00:00 2024-09-21T21:31:13.574 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 234 kB/s | 26 kB 00:00 2024-09-21T21:31:13.633 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 293 kB/s | 49 kB 00:00 2024-09-21T21:31:13.683 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-21T21:31:13.708 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-21T21:31:13.743 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 913 kB/s | 54 kB 00:00 2024-09-21T21:31:13.775 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T21:31:13.801 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-21T21:31:13.835 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.8 MB/s | 220 kB 00:00 2024-09-21T21:31:13.868 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-09-21T21:31:13.893 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.0 MB/s | 279 kB 00:00 2024-09-21T21:31:13.910 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-21T21:31:13.952 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T21:31:13.953 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:13.954 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.6 MB/s | 2.2 MB 00:00 2024-09-21T21:31:14.019 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:14.036 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:14.036 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:14.135 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:14.138 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:14.658 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:14.840 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T21:31:14.896 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T21:31:14.952 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T21:31:14.987 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T21:31:15.021 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T21:31:15.135 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T21:31:15.254 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T21:31:15.307 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T21:31:16.067 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T21:31:16.111 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T21:31:16.144 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T21:31:16.193 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T21:31:16.222 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T21:31:17.062 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T21:31:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T21:31:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T21:31:17.063 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T21:31:17.406 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T21:31:17.407 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T21:31:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T21:31:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T21:31:17.408 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:17.408 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:17.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T21:31:18.299 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:57 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:18.439 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:18.439 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:18.439 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:31:18.439 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-09-21T21:31:18.440 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:18.441 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 7.0 M 2024-09-21T21:31:18.441 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-09-21T21:31:18.441 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:18.820 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-21T21:31:18.928 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 724 kB/s | 154 kB 00:00 2024-09-21T21:31:18.928 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T21:31:18.928 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.9 MB/s | 548 kB 00:00 2024-09-21T21:31:18.954 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-21T21:31:19.046 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-21T21:31:19.048 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:19.049 INFO:teuthology.orchestra.run.smithi063.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-21T21:31:19.122 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:19.133 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:19.133 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:19.240 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:19.240 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:19.558 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:19.728 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T21:31:19.774 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T21:31:20.324 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T21:31:20.412 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T21:31:20.488 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T21:31:20.518 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:31:21.276 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:31:21.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-21T21:31:21.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T21:31:21.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T21:31:21.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T21:31:21.277 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T21:31:21.556 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T21:31:21.557 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T21:31:21.557 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T21:31:21.557 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T21:31:21.557 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:21.557 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:21.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-09-21T21:31:22.428 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:01 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:22.566 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:22.566 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:22.566 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-09-21T21:31:22.567 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:22.568 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 963 k 2024-09-21T21:31:22.568 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 4.1 M 2024-09-21T21:31:22.568 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:22.855 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 430 kB/s | 80 kB 00:00 2024-09-21T21:31:22.939 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.2 MB/s | 883 kB 00:00 2024-09-21T21:31:22.939 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:22.939 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.5 MB/s | 963 kB 00:00 2024-09-21T21:31:22.951 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:22.976 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:22.976 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:23.062 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:23.063 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:23.267 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:23.479 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T21:31:23.558 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:31:24.256 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:31:24.256 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:24.572 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:24.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-volume 2024-09-21T21:31:25.445 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:04 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:25.583 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:25.584 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:25.585 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 285 k 2024-09-21T21:31:25.585 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 1.3 M 2024-09-21T21:31:25.585 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:25.809 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T21:31:25.810 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:25.810 INFO:teuthology.orchestra.run.smithi063.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T21:31:25.811 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:25.820 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:25.820 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:25.852 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:25.853 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:26.089 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:26.153 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:26.170 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:26.170 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:31:26.170 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T21:31:26.170 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:27.166 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:27.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-09-21T21:31:28.072 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:07 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:28.213 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:28.214 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:28.214 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 125 k 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 445 k 2024-09-21T21:31:28.215 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:28.430 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 580 kB/s | 125 kB 00:00 2024-09-21T21:31:28.431 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:28.431 INFO:teuthology.orchestra.run.smithi063.stdout:Total 575 kB/s | 125 kB 00:00 2024-09-21T21:31:28.432 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:28.437 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:28.438 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:28.505 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:28.505 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:28.663 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:28.773 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:29.235 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:29.482 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:29.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:29.482 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:29.482 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:29.483 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:29.483 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:29.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-09-21T21:31:30.355 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:09 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:30.448 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:30.494 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:30.495 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:30.495 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:30.567 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-09-21T21:31:31.179 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:10 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:31.321 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:31.321 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 32 k 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 142 k 2024-09-21T21:31:31.322 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:31.483 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 197 kB/s | 32 kB 00:00 2024-09-21T21:31:31.484 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:31.484 INFO:teuthology.orchestra.run.smithi063.stdout:Total 195 kB/s | 32 kB 00:00 2024-09-21T21:31:31.484 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:31.488 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:31.488 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:31.505 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:31.505 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:31.591 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:31.725 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:32.089 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:32.336 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:32.337 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:32.337 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:32.337 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:32.337 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:32.337 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:32.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-09-21T21:31:33.108 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:12 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:33.199 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:33.244 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:33.245 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:33.245 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:33.318 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-09-21T21:31:33.921 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:12 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:34.013 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:34.062 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:34.063 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:34.063 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:34.133 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-09-21T21:31:34.744 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:13 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:34.837 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:34.883 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:34.884 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:34.884 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:34.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-09-21T21:31:35.558 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:14 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:35.650 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:35.696 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:35.697 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:35.698 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:35.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-09-21T21:31:36.373 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:15 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:36.465 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:36.511 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:36.512 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:36.512 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:36.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-09-21T21:31:37.190 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:16 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:37.282 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:31:37.328 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:37.329 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-09-21T21:31:37.329 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:37.410 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-09-21T21:31:38.015 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:17 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:38.154 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:38.154 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:38.154 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-21T21:31:38.154 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:38.154 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 91 k 2024-09-21T21:31:38.155 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 321 k 2024-09-21T21:31:38.156 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:38.353 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 461 kB/s | 91 kB 00:00 2024-09-21T21:31:38.353 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:38.354 INFO:teuthology.orchestra.run.smithi063.stdout:Total 458 kB/s | 91 kB 00:00 2024-09-21T21:31:38.354 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:38.363 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:38.364 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:38.431 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:38.431 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:38.583 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:38.687 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:39.068 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:39.328 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:39.647 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-09-21T21:31:40.261 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:19 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:40.400 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:40.401 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:40.401 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-09-21T21:31:40.401 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:40.402 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:40.403 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:40.403 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.5 M 2024-09-21T21:31:40.403 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 30 M 2024-09-21T21:31:40.403 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:40.739 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-21T21:31:40.739 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:40.740 INFO:teuthology.orchestra.run.smithi063.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-21T21:31:40.740 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:40.753 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:40.753 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:40.841 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:40.841 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:41.453 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:41.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:31:41.558 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:42.836 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:43.068 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-09-21T21:31:43.683 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:02:22 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:31:43.825 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-09-21T21:31:43.826 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:43.827 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 176 k 2024-09-21T21:31:43.827 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 685 k 2024-09-21T21:31:43.827 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:31:44.033 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 853 kB/s | 176 kB 00:00 2024-09-21T21:31:44.034 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:31:44.034 INFO:teuthology.orchestra.run.smithi063.stdout:Total 847 kB/s | 176 kB 00:00 2024-09-21T21:31:44.034 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:31:44.045 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:31:44.045 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:31:44.114 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:31:44.114 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:31:44.279 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:31:44.385 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:44.818 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:45.054 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:31:45.334 DEBUG:teuthology.parallel:result is None 2024-09-21T21:31:45.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:31:45.492 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T21:31:45.516 INFO:teuthology.orchestra.run.smithi063.stdout:19.3.0-5092.g62da54b4.el9 2024-09-21T21:31:45.517 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-21T21:31:45.517 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-21T21:31:45.519 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T21:31:45.520 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:45.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T21:31:45.585 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T21:31:45.586 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:45.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T21:31:45.653 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T21:31:45.720 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T21:31:45.720 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:45.720 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T21:31:45.786 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T21:31:45.853 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T21:31:45.853 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:45.853 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T21:31:45.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T21:31:45.994 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T21:31:46.084 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T21:31:46.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2024-09-21T21:31:46.112 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T21:31:46.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T21:31:46.178 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T21:31:46.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T21:31:46.254 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T21:31:46.255 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T21:31:46.255 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_rand_read}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914352', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi063.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFwD5kKDz+RybxrtKyJnzvWJkKwZzaktx4IwSY0JNT1hEzZMe/q4CDC+18ssmfeNfvyPVrOhS6fh9BBH1VazQIQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T21:31:46.256 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T21:31:46.300 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T21:31:46.367 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:46.368 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T21:31:46.424 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T21:31:46.425 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-09-21T21:31:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T21:31:46.480 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:46.480 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T21:31:46.480 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:46.480 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:46.481 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-21 21:31:41.834848067 +0000 2024-09-21T21:31:46.481 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-21 21:28:40.844774122 +0000 2024-09-21T21:31:46.481 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-21 21:28:40.844774122 +0000 2024-09-21T21:31:46.481 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-21 21:28:40.844774122 +0000 2024-09-21T21:31:46.481 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T21:31:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-21T21:31:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-21T21:31:46.546 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.0001898 s, 2.7 MB/s 2024-09-21T21:31:46.547 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T21:31:46.604 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-09-21T21:31:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T21:31:46.660 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:46.660 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-21T21:31:46.660 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:46.660 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:46.661 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-21 21:31:41.835848058 +0000 2024-09-21T21:31:46.661 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-21 21:28:41.182770857 +0000 2024-09-21T21:31:46.661 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-21 21:28:41.182770857 +0000 2024-09-21T21:31:46.661 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-21 21:28:41.182770857 +0000 2024-09-21T21:31:46.661 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T21:31:46.725 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-21T21:31:46.725 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-21T21:31:46.725 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.0002014 s, 2.5 MB/s 2024-09-21T21:31:46.726 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T21:31:46.783 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-09-21T21:31:46.839 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T21:31:46.839 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-21 21:31:41.835848058 +0000 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-21 21:28:41.513767659 +0000 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-21 21:28:41.513767659 +0000 2024-09-21T21:31:46.840 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-21 21:28:41.513767659 +0000 2024-09-21T21:31:46.840 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T21:31:46.903 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-21T21:31:46.903 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-21T21:31:46.903 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000179392 s, 2.9 MB/s 2024-09-21T21:31:46.904 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T21:31:46.960 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-09-21 21:31:41.836848048 +0000 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-09-21 21:28:41.836764539 +0000 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-09-21 21:28:41.836764539 +0000 2024-09-21T21:31:47.016 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-09-21 21:28:41.836764539 +0000 2024-09-21T21:31:47.017 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T21:31:47.080 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-09-21T21:31:47.080 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-09-21T21:31:47.081 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000192306 s, 2.7 MB/s 2024-09-21T21:31:47.082 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T21:31:47.138 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:47.138 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:31:47.138 INFO:tasks.ceph:Generating config... 2024-09-21T21:31:47.139 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-21T21:31:47.139 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-21T21:31:47.140 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-21T21:31:47.141 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] auth client required = none 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] auth service required = none 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] auth supported = none 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-21T21:31:47.141 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-21T21:31:47.142 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-21T21:31:47.143 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-21T21:31:47.144 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-21T21:31:47.145 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-21T21:31:47.146 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-21T21:31:47.147 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-21T21:31:47.147 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-21T21:31:47.147 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-21T21:31:47.147 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-21T21:31:47.148 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-21T21:31:47.148 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-21T21:31:47.148 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-21T21:31:47.148 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T21:31:47.148 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T21:31:47.233 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T21:31:47.237 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T21:31:47.289 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:31:47.315 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.63')] 2024-09-21T21:31:47.315 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.63', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-21T21:31:47.316 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:47.316 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:31:47.370 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.63 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:47.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:31:47.462+0000 7f3f01716f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:47.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:31:47.462+0000 7f3f01716f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:47.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:31:47.462+0000 7f3f01716f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid f6b27aae-fb66-4d0b-bb72-52764dd0bc6f 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = quincy 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:fsid f6b27aae-fb66-4d0b-bb72-52764dd0bc6f 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2024-09-21T21:31:47.463797+0000 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:created 2024-09-21T21:31:47.463797+0000 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 17 (quincy) 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:election_strategy: 1 2024-09-21T21:31:47.465 INFO:teuthology.orchestra.run.smithi063.stdout:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2024-09-21T21:31:47.466 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-21T21:31:47.467 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:31:47.482 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f6b27aae-fb66-4d0b-bb72-52764dd0bc6f... 2024-09-21T21:31:47.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = none 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2024-09-21T21:31:47.568 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2024-09-21T21:31:47.569 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.570 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = 172.21.15.63 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: auth client required = none 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: auth cluster required = none 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: auth service required = none 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug asok = 0/0 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 0/0 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug bdev = 0/0 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluefs = 0/0 2024-09-21T21:31:47.571 INFO:teuthology.orchestra.run.smithi063.stdout: debug bluestore = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug buffer = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug civetweb = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug client = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug compressor = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug context = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug crush = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug crypto = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug dpdk = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug eventtrace = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug filer = 0/0 2024-09-21T21:31:47.572 INFO:teuthology.orchestra.run.smithi063.stdout: debug filestore = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug finisher = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug fuse = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug heartbeatmap = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug javaclient = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug journal = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug journaler = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug kstore = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug leveldb = 0/0 2024-09-21T21:31:47.573 INFO:teuthology.orchestra.run.smithi063.stdout: debug lockdep = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds balancer = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds locker = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds log = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds log expire = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug mds migrator = 0/0 2024-09-21T21:31:47.574 INFO:teuthology.orchestra.run.smithi063.stdout: debug memdb = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgrc = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug objclass = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug objectcacher = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug objecter = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug optracker = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 0/0 2024-09-21T21:31:47.575 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug perfcounter = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rados = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rbd = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rbd mirror = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rbd replay = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug refs = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw lifecycle = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw notification = 0/0 2024-09-21T21:31:47.576 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw sync = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: debug rocksdb = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: debug striper = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: debug throttle = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: debug timer = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: debug tp = 0/0 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms client mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms cluster mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms mon client mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms mon cluster mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms mon service mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: ms service mode = crc 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = f6b27aae-fb66-4d0b-bb72-52764dd0bc6f 2024-09-21T21:31:47.577 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd mclock profile = high_recovery_ops 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2024-09-21T21:31:47.578 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug trim objects = true 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2024-09-21T21:31:47.579 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug monc = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: crimson alien thread cpu cores = 6-7 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: crimson osd obc lru size = 10 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug filestore = 0/0 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug journal = 0/0 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 0/0 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 0/0 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore backref = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore cache = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore cleaner = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore device = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore epm = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore fixedkv tree = 20 2024-09-21T21:31:47.580 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore journal = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore lba = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore odata = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore ompap = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore onode = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore t = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug seastore tm = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: osd objectstore = seastore 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: osd shutdown pgref assert = True 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2024-09-21T21:31:47.581 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 0/0 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 0/0 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 0/0 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2024-09-21T21:31:47.582 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: # 1m isn't quite enough 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: mon_down_mkfs_grace = 2m 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.583 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_filestore_osds = false 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default crimson = True 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-09-21T21:31:47.584 INFO:teuthology.orchestra.run.smithi063.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.3] 2024-09-21T21:31:47.585 INFO:teuthology.orchestra.run.smithi063.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:31:47.586 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T21:31:47.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T21:31:47.653 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T21:31:47.654 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:47.654 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T21:31:47.669 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:47.669 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T21:31:47.725 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:31:47.725 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:47.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T21:31:47.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:31:47.802 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:31:47.802 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:47.857 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T21:31:47.858 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T21:31:47.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:31:47.958 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:31:47.962 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T21:31:47.962 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T21:31:47.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T21:31:48.013 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T21:31:48.026 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T21:31:48.026 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:31:48.027 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:48.052 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:48.052 INFO:tasks.ceph:role: osd.0 2024-09-21T21:31:48.052 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:31:48.053 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T21:31:48.125 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:48.125 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:48.125 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:48.126 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:48.631 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-21T21:31:48.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-21T21:31:48.634 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:48.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:48.816 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:48.884 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:48.885 INFO:tasks.ceph:role: osd.1 2024-09-21T21:31:48.885 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:31:48.885 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T21:31:48.959 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:48.959 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:48.959 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:48.959 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:48.960 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:49.414 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-21T21:31:49.416 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-21T21:31:49.417 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:49.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:49.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:49.607 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:31:49.607 INFO:tasks.ceph:role: osd.2 2024-09-21T21:31:49.607 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:31:49.607 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T21:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:31:49.681 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:31:49.682 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:31:50.176 INFO:teuthology.orchestra.run.smithi063.stdout:Discarding blocks...Done. 2024-09-21T21:31:50.178 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2024-09-21T21:31:50.179 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:50.332 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T21:31:50.366 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:50.676 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:50.677 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:50,677 seastar - Reactor backend: linux-aio 2024-09-21T21:31:50.690 INFO:teuthology.orchestra.run.smithi063.stderr:==28093==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:50.830 INFO:teuthology.orchestra.run.smithi063.stderr:DEBUG 2024-09-21 21:31:50,829 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:51.367 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:31:51.421 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:51.425 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:51,425 seastar - Reactor backend: linux-aio 2024-09-21T21:31:51.435 INFO:teuthology.orchestra.run.smithi063.stderr:==28091==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:51.500 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10d778 0xdd31a 0x4ab1c 0x3612f03 0x3629204 0x382a100 0x416e695 0x5f9bd94 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:51.500 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:51.740 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xba1c8d3 0xb9bcc48 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.740 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x36074cd 0x3b8bf66 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9e567f 0xba14771 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.740 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:51.740 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:51.798 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42198 0x3ef7b 0xfd531 0xb83682d 0xba16ccc 0xba16dfe 0xba16f7a 0xba17729 0xb9b1eb4 0x9826b31 0x9a94970 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.798 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42198 0x3ecea 0xfc9b1 0x3869141 0xba0df79 0xba0e13c 0xba0e2a9 0xb9b5385 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a9320e 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.798 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8e08c7a4 0xffffffff8ecd1db2 0xffffffff8ee00bb2 2024-09-21T21:31:51.798 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8e1f603f 0xffffffff8e1f64c0 0xffffffff8e0908cb 0xffffffff8e3a8199 0xffffffff8e38e203 0xffffffff8e39699b 0xffffffff8e396e1c 0xffffffff8e08c884 0xffffffff8ecd1db2 0xffffffff8ee00bb2 2024-09-21T21:31:51.897 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4b14c 0x95c7184 0x9add047 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.898 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:51.955 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41e8e 0x4214e 0x3ef7b 0xfd531 0xb83682d 0xba16ccc 0xba16dfe 0xba16f7a 0xba17729 0xb9b1eb4 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:51.955 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.013 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524fcb 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:52.013 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.070 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524fd9 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:52.071 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.093 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abc8 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:52.093 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.093 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253a6 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:52.093 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.150 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525170 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:52.150 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525d15 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.258 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133ccf 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:52.258 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:52.606 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10252a 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:52.607 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:53.638 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:31:53.639 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:31:53.639 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:53.639 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:53.639 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:31:53.667 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T21:31:53.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:53.987 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:53.988 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:53,987 seastar - Reactor backend: linux-aio 2024-09-21T21:31:53.993 INFO:teuthology.orchestra.run.smithi063.stderr:==28140==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:54.101 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3814dec 0x5de3fea 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:54.101 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:54.134 INFO:teuthology.orchestra.run.smithi063.stderr:DEBUG 2024-09-21 21:31:54,133 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:54.665 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:31:54.721 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:54.725 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:54,725 seastar - Reactor backend: linux-aio 2024-09-21T21:31:54.736 INFO:teuthology.orchestra.run.smithi063.stderr:==28138==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:54.800 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x381a0d7 0x601ba36 0x601bc69 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:54.801 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.039 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4218f 0x3ecea 0xfc9b1 0x3869141 0xb9f36a3 0xb9b1ecf 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.039 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4bebf 0xb9e7536 0xb9e8829 0xba170df 0xba17729 0xb9b1eb4 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.039 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.040 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.097 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9cd8bb 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.097 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb836bbb 0xb836d56 0xb837301 0xb83770c 0xb8379cb 0xb838488 0xba09156 0xb9ae311 0x9a94eb7 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.097 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.097 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.098 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x36074ba 0x3b8bf66 0x3b8c15e 0xb9d16cf 0xb9d2155 0xba1750a 0xb9b1eb4 0x9826b31 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:55.098 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.156 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x15979c 0xc928c 0xc9e39 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:55.156 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.198 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 66 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x387c4e3 0x9bff8d2 0x9adc9b5 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.198 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.256 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a1cf 0xfe604 0x5ca6522 0x5ca66fc 0x5ca69b3 0x5ca6978 0x5ca6aa8 0x961a19c 0x9add627 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.256 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.397 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x552576d 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.397 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.397 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55251a0 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.397 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x552518c 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.454 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.562 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd2cf 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:55.562 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:55.944 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55251dc 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:55.945 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:56.002 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d18 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:56.002 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:56.902 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:31:56.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T21:31:56.955 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:31:57.260 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:57.261 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:57,261 seastar - Reactor backend: linux-aio 2024-09-21T21:31:57.267 INFO:teuthology.orchestra.run.smithi063.stderr:==28187==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:57.375 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5da4dcf 0x5fcceb9 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:57.375 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:57.409 INFO:teuthology.orchestra.run.smithi063.stderr:DEBUG 2024-09-21 21:31:57,408 [shard 0:main] seastar - reactor::drain 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:31:57.941 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:31:57.998 INFO:teuthology.orchestra.run.smithi063.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:31:58.002 INFO:teuthology.orchestra.run.smithi063.stderr:INFO 2024-09-21 21:31:58,002 seastar - Reactor backend: linux-aio 2024-09-21T21:31:58.013 INFO:teuthology.orchestra.run.smithi063.stderr:==28185==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:31:58.077 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41e88 0x4214e 0x3ef7b 0xfd531 0x361a443 0x361a64c 0x36292ab 0x382a100 0x416e695 0x5f9bd94 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:58.078 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.317 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x98281f3 0xb82f714 0xb9bef04 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.318 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.318 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42198 0x3ecea 0xfc9b1 0x3869141 0xb836412 0xb83655a 0xb836643 0xb836903 0xb8369f6 0xb836ad6 0xb836d56 0xb837301 0xb83770c 0xb8379cb 0xba09729 0xba0b737 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.318 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8e68c7ad 0xffffffff8e39b83c 0xffffffff8e08c811 0xffffffff8ecd1db2 0xffffffff8ee00bb2 2024-09-21T21:31:58.375 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3ac7364 0x3b8c153 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.375 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3779709 0x3ac727c 0x3b8c153 0xb9e0fae 0xba1c4c3 0xba1c84e 0xb9bcc48 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.375 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.375 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.401 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41f57 0x4214e 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb837196 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xb9bdf19 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:58.401 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5da4d84 0x5fcce85 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5df61a3 0x5df7058 0x378adb3 0x378b066 0x396f348 0xb78e558 0xb0db2c2 0xb0d90e6 0x992938b 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:58.459 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.579 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abef 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:58.579 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.636 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524e80 0x5525e58 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:31:58.637 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.687 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524fcb 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.687 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.745 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab10 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.745 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.753 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d0c 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.753 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4aaff 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.853 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:31:58.860 INFO:teuthology.orchestra.run.smithi063.stderr:Reactor stalled for 230 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525e71 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-21T21:31:58.860 INFO:teuthology.orchestra.run.smithi063.stderr:kernel callstack: 2024-09-21T21:32:00.235 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:32:00.235 INFO:teuthology.orchestra.run.smithi063.stderr:Suppressions used: 2024-09-21T21:32:00.235 INFO:teuthology.orchestra.run.smithi063.stderr: count bytes template 2024-09-21T21:32:00.235 INFO:teuthology.orchestra.run.smithi063.stderr: 1 8 MallocExtension::Register 2024-09-21T21:32:00.235 INFO:teuthology.orchestra.run.smithi063.stderr:----------------------------------------------------- 2024-09-21T21:32:00.236 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:32:00.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T21:32:00.290 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T21:32:00.291 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:00.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T21:32:00.356 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:00.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T21:32:00.422 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:00.423 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T21:32:00.488 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:00.488 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T21:32:00.554 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:00.554 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T21:32:00.611 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T21:32:00.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T21:32:00.674 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBDO+9mgpEcORAADvrSj7918tnqLquLTfLdvA== 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBHO+9meqkvIxAAwDljBX798bJwjD04DoZbSg== 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBKO+9mpKAeNRAAhbOC6QqhxAM3QQ2EjqDErw== 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBOO+9m0mg9ChAAhWn8pMwvtb6nPHqFSkzWjA== 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-09-21T21:32:00.675 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQBEO+9mNEW+ABAAEwE+jmCNCjmrfmh/LhtvkQ== 2024-09-21T21:32:00.676 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T21:32:00.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:32:00.843 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:32:00.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:32:00.969 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T21:32:01.035 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T21:32:01.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T21:32:01.062 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T21:32:01.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.173+0000 7f00a8367c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.175 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.173+0000 7f00a8367c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T21:32:01.213 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:32:01.269 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T21:32:01.269 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T21:32:01.269 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T21:32:01.311 INFO:tasks.ceph.mon.a:Started 2024-09-21T21:32:01.311 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T21:32:01.311 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T21:32:01.311 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T21:32:01.314 INFO:tasks.ceph.mgr.x:Started 2024-09-21T21:32:01.314 DEBUG:tasks.ceph:set 0 configs 2024-09-21T21:32:01.314 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T21:32:01.314 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T21:32:01.399 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-09-21T21:32:01.397+0000 7f8caee36c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.399 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-09-21T21:32:01.397+0000 7f8caee36c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.400 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-09-21T21:32:01.397+0000 7f8caee36c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.400 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-21T21:32:01.398+0000 7fd49ae69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.430 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.422+0000 7f8d554bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.432 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.430+0000 7f8d554bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.502 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-09-21T21:32:01.514 INFO:tasks.ceph:check_enable_crimson: True 2024-09-21T21:32:01.514 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-21T21:32:01.515 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-21T21:32:01.604 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-21T21:32:01.602+0000 7fd49ae69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.604 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-21T21:32:01.602+0000 7fd49ae69000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.619+0000 7f545df91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:01.626+0000 7f545df91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:01.728 INFO:tasks.ceph.mon.a.smithi063.stderr:2024-09-21T21:32:01.726+0000 7f8ca57f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-21T21:32:01.728 INFO:tasks.ceph.mon.a.smithi063.stderr:Please be aware that this feature is experimental, untested, 2024-09-21T21:32:01.728 INFO:tasks.ceph.mon.a.smithi063.stderr:unsupported, and may result in data corruption, data loss, 2024-09-21T21:32:01.728 INFO:tasks.ceph.mon.a.smithi063.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-21T21:32:01.728 INFO:tasks.ceph.mon.a.smithi063.stderr:feature with important data. 2024-09-21T21:32:01.729 INFO:tasks.ceph.mon.a.smithi063.stderr: 2024-09-21T21:32:02.515 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T21:32:02.516 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:02.516 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T21:32:02.541 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:02.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T21:32:02.608 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:02.608 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T21:32:02.674 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7 0 2024-09-21T21:32:02.825 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:02.817+0000 7fc3c1d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:02.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:02.825+0000 7fc3c1d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:02.905 INFO:teuthology.orchestra.run.smithi063.stdout:0 2024-09-21T21:32:02.917 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new bb8e5833-e047-40be-b944-2cce72182891 1 2024-09-21T21:32:03.030 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.023+0000 7fdb02191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.031+0000 7fdb02191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.120 INFO:teuthology.orchestra.run.smithi063.stdout:1 2024-09-21T21:32:03.132 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd new a21faece-c073-44a2-8f94-6abb1856b416 2 2024-09-21T21:32:03.244 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.237+0000 7f057244f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.246 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.245+0000 7f057244f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.342 INFO:teuthology.orchestra.run.smithi063.stdout:2 2024-09-21T21:32:03.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-21T21:32:03.386 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T21:32:03.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-21T21:32:03.429 INFO:tasks.ceph.osd.0:Started 2024-09-21T21:32:03.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-21T21:32:03.459 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T21:32:03.459 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-21T21:32:03.502 INFO:tasks.ceph.osd.1:Started 2024-09-21T21:32:03.502 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-21T21:32:03.537 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T21:32:03.537 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-21T21:32:03.580 INFO:tasks.ceph.osd.2:Started 2024-09-21T21:32:03.580 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:03.705 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.698+0000 7fba64151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.707 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:03.706+0000 7fba64151640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:03.775 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:03.776 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":6,"fsid":"f6b27aae-fb66-4d0b-bb72-52764dd0bc6f","created":"2024-09-21T21:32:01.422892+0000","modified":"2024-09-21T21:32:03.340836+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:32:03.340836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb8e5833-e047-40be-b944-2cce72182891","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a21faece-c073-44a2-8f94-6abb1856b416","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:03.790 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T21:32:03.791 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T21:32:04.092 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:04.199 INFO:teuthology.misc.health.smithi063.stderr:2024-09-21T21:32:04.194+0000 7f20f906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:04.201 INFO:teuthology.misc.health.smithi063.stderr:2024-09-21T21:32:04.200+0000 7f20f906c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:04.271 INFO:teuthology.misc.health.smithi063.stdout: 2024-09-21T21:32:04.271 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":6,"fsid":"f6b27aae-fb66-4d0b-bb72-52764dd0bc6f","created":"2024-09-21T21:32:01.422892+0000","modified":"2024-09-21T21:32:03.340836+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:32:03.340836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bb8e5833-e047-40be-b944-2cce72182891","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a21faece-c073-44a2-8f94-6abb1856b416","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:04.283 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T21:32:07.278 INFO:tasks.ceph.mgr.x.smithi063.stderr:2024-09-21T21:32:07.276+0000 7fd438617640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T21:32:10.585 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:10.699 INFO:teuthology.misc.health.smithi063.stderr:2024-09-21T21:32:10.693+0000 7f72cc05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:10.702 INFO:teuthology.misc.health.smithi063.stderr:2024-09-21T21:32:10.701+0000 7f72cc05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:11.119 INFO:teuthology.misc.health.smithi063.stdout: 2024-09-21T21:32:11.119 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":10,"fsid":"f6b27aae-fb66-4d0b-bb72-52764dd0bc6f","created":"2024-09-21T21:32:01.422892+0000","modified":"2024-09-21T21:32:10.336682+0000","last_up_change":"2024-09-21T21:32:10.336682+0000","last_in_change":"2024-09-21T21:32:03.340836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":1663049632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1663049632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":1663049632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1663049632}]},"public_addr":"172.21.15.63:6807/1663049632","cluster_addr":"172.21.15.63:6806/1663049632","heartbeat_back_addr":"172.21.15.63:6819/1663049632","heartbeat_front_addr":"172.21.15.63:6818/1663049632","state":["exists","up"]},{"osd":1,"uuid":"bb8e5833-e047-40be-b944-2cce72182891","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":3041388459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":3041388459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":3041388459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3041388459}]},"public_addr":"172.21.15.63:6803/3041388459","cluster_addr":"172.21.15.63:6802/3041388459","heartbeat_back_addr":"172.21.15.63:6813/3041388459","heartbeat_front_addr":"172.21.15.63:6812/3041388459","state":["exists","up"]},{"osd":2,"uuid":"a21faece-c073-44a2-8f94-6abb1856b416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":4252157986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":4252157986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6823","nonce":4252157986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4252157986}]},"public_addr":"172.21.15.63:6808/4252157986","cluster_addr":"172.21.15.63:6809/4252157986","heartbeat_back_addr":"172.21.15.63:6823/4252157986","heartbeat_front_addr":"172.21.15.63:6824/4252157986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:11.139 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T21:32:11.140 INFO:tasks.ceph:Creating RBD pool 2024-09-21T21:32:11.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T21:32:11.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:11.253+0000 7fabc8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:11.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:11.265+0000 7fabc8d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:12.352 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'rbd' created 2024-09-21T21:32:12.375 DEBUG:teuthology.orchestra.run.smithi063:> rbd --cluster ceph pool init rbd 2024-09-21T21:32:12.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:12.602+0000 7fe75b020ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:12.604 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:12.602+0000 7fe75b020ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:12.607 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:12.606+0000 7fe75b020ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:15.434 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T21:32:15.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T21:32:15.435 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T21:32:15.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:15.544+0000 7f620bca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:15.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:15.551+0000 7f620bca3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:15.752 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:15.765 INFO:teuthology.orchestra.run.smithi063.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-21T21:32:15.766 INFO:tasks.ceph_manager:config epoch is 1 2024-09-21T21:32:15.766 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T21:32:15.766 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T21:32:15.766 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T21:32:15.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:15.874+0000 7f8362c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:15.881 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:15.880+0000 7f8362c4f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.133 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:16.145 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":3919061479},{"type":"v1","addr":"172.21.15.63:6801","nonce":3919061479}]},"active_addr":"172.21.15.63:6801/3919061479","active_change":"2024-09-21T21:32:06.263748+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3727528233}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":549298018}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2065434167}]}]} 2024-09-21T21:32:16.148 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T21:32:16.148 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T21:32:16.149 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:16.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.257+0000 7fbeed191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.262+0000 7fbeed191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.462 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:16.462 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":14,"fsid":"f6b27aae-fb66-4d0b-bb72-52764dd0bc6f","created":"2024-09-21T21:32:01.422892+0000","modified":"2024-09-21T21:32:15.358243+0000","last_up_change":"2024-09-21T21:32:10.336682+0000","last_in_change":"2024-09-21T21:32:03.340836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:32:11.484842+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:32:12.288464+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":1663049632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1663049632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":1663049632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1663049632}]},"public_addr":"172.21.15.63:6807/1663049632","cluster_addr":"172.21.15.63:6806/1663049632","heartbeat_back_addr":"172.21.15.63:6819/1663049632","heartbeat_front_addr":"172.21.15.63:6818/1663049632","state":["exists","up"]},{"osd":1,"uuid":"bb8e5833-e047-40be-b944-2cce72182891","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":3041388459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":3041388459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":3041388459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3041388459}]},"public_addr":"172.21.15.63:6803/3041388459","cluster_addr":"172.21.15.63:6802/3041388459","heartbeat_back_addr":"172.21.15.63:6813/3041388459","heartbeat_front_addr":"172.21.15.63:6812/3041388459","state":["exists","up"]},{"osd":2,"uuid":"a21faece-c073-44a2-8f94-6abb1856b416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":4252157986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":4252157986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6823","nonce":4252157986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4252157986}]},"public_addr":"172.21.15.63:6808/4252157986","cluster_addr":"172.21.15.63:6809/4252157986","heartbeat_back_addr":"172.21.15.63:6823/4252157986","heartbeat_front_addr":"172.21.15.63:6824/4252157986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:16.475 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T21:32:16.475 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:32:16.594 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.588+0000 7f86fcad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.596 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.594+0000 7f86fcad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:16.796 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":14,"fsid":"f6b27aae-fb66-4d0b-bb72-52764dd0bc6f","created":"2024-09-21T21:32:01.422892+0000","modified":"2024-09-21T21:32:15.358243+0000","last_up_change":"2024-09-21T21:32:10.336682+0000","last_in_change":"2024-09-21T21:32:03.340836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:32:11.484842+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:32:12.288464+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a33551d1-58e4-4ab1-a6b9-dbfc4bbf5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6807","nonce":1663049632}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1663049632}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6819","nonce":1663049632}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1663049632}]},"public_addr":"172.21.15.63:6807/1663049632","cluster_addr":"172.21.15.63:6806/1663049632","heartbeat_back_addr":"172.21.15.63:6819/1663049632","heartbeat_front_addr":"172.21.15.63:6818/1663049632","state":["exists","up"]},{"osd":1,"uuid":"bb8e5833-e047-40be-b944-2cce72182891","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6803","nonce":3041388459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":3041388459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6813","nonce":3041388459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":3041388459}]},"public_addr":"172.21.15.63:6803/3041388459","cluster_addr":"172.21.15.63:6802/3041388459","heartbeat_back_addr":"172.21.15.63:6813/3041388459","heartbeat_front_addr":"172.21.15.63:6812/3041388459","state":["exists","up"]},{"osd":2,"uuid":"a21faece-c073-44a2-8f94-6abb1856b416","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":4252157986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6809","nonce":4252157986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6823","nonce":4252157986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":4252157986}]},"public_addr":"172.21.15.63:6808/4252157986","cluster_addr":"172.21.15.63:6809/4252157986","heartbeat_back_addr":"172.21.15.63:6823/4252157986","heartbeat_front_addr":"172.21.15.63:6824/4252157986","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:32:16.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T21:32:16.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T21:32:16.810 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T21:32:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.923+0000 7fae63d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.923+0000 7f07b9391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.923+0000 7f2c82b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.931+0000 7fae63d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.931+0000 7f07b9391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:16.931+0000 7f2c82b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:16.962 INFO:teuthology.orchestra.run.smithi063.stdout:42949672966 2024-09-21T21:32:16.962 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:16.963 INFO:teuthology.orchestra.run.smithi063.stdout:34359738376 2024-09-21T21:32:16.963 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:16.965 INFO:teuthology.orchestra.run.smithi063.stdout:42949672966 2024-09-21T21:32:16.966 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:17.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.074+0000 7f2315b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.082+0000 7f2315b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.125 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.118+0000 7f7220148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.127 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.126+0000 7f7220148640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.128 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.121+0000 7f2504d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.130 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:17.129+0000 7f2504d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:17.310 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:17.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-21T21:32:17.333 INFO:teuthology.orchestra.run.smithi063.stdout:34359738372 2024-09-21T21:32:17.343 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:17.347 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-21T21:32:17.356 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-21T21:32:18.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:18.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:18.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:18.457 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.448+0000 7f3d6667d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.458+0000 7f3d6667d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.496 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.490+0000 7f58918a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.498 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.497+0000 7f58918a3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.499 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.493+0000 7f585b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.502 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:18.500+0000 7f585b991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:18.677 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:18.690 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-21T21:32:18.708 INFO:teuthology.orchestra.run.smithi063.stdout:34359738372 2024-09-21T21:32:18.715 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:18.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-21T21:32:18.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-21T21:32:19.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:19.723 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:19.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:19.827 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.818+0000 7f779fb48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:19.829 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.828+0000 7f779fb48640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:19.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.857+0000 7fc1ce711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:19.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.858+0000 7f9933391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:19.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.865+0000 7fc1ce711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:19.866 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:19.865+0000 7f9933391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:20.037 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:20.050 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-21T21:32:20.063 INFO:teuthology.orchestra.run.smithi063.stdout:34359738372 2024-09-21T21:32:20.081 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-21T21:32:20.085 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:20.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-21T21:32:21.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:21.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:32:21.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:21.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.172+0000 7fd3ae646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.180+0000 7fd3ae646640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.223 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.215+0000 7f2f82191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.225 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.224+0000 7f2f82191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.252+0000 7fbb6364f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:21.259+0000 7fbb6364f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:21.397 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:21.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-21T21:32:21.437 INFO:teuthology.orchestra.run.smithi063.stdout:34359738377 2024-09-21T21:32:21.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738377 for osd.1 2024-09-21T21:32:21.452 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:21.481 INFO:teuthology.orchestra.run.smithi063.stdout:42949672964 2024-09-21T21:32:21.495 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-21T21:32:22.412 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:32:22.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:32:22.544 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.537+0000 7fd566951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:22.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.544+0000 7fd566951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:22.616 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.610+0000 7f267cb08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:22.619 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.617+0000 7f267cb08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:22.765 INFO:teuthology.orchestra.run.smithi063.stdout:42949672969 2024-09-21T21:32:22.779 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.2 2024-09-21T21:32:22.779 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:22.836 INFO:teuthology.orchestra.run.smithi063.stdout:42949672969 2024-09-21T21:32:22.849 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.0 2024-09-21T21:32:22.850 DEBUG:teuthology.parallel:result is None 2024-09-21T21:32:22.850 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T21:32:22.850 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:32:22.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.959+0000 7fa037751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:22.966 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:22.964+0000 7fa037751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:23.161 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-21T21:32:23.173 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-21T21:32:22.270312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.786621"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.389269+0000","last_change":"2024-09-21T21:32:15.395284+0000","last_active":"2024-09-21T21:32:15.389269+0000","last_peered":"2024-09-21T21:32:15.389269+0000","last_clean":"2024-09-21T21:32:15.389269+0000","last_became_active":"2024-09-21T21:32:13.513751+0000","last_became_peered":"2024-09-21T21:32:13.513751+0000","last_unstale":"2024-09-21T21:32:15.389269+0000","last_undegraded":"2024-09-21T21:32:15.389269+0000","last_fullsized":"2024-09-21T21:32:15.389269+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.385555+0000","last_change":"2024-09-21T21:32:15.391507+0000","last_active":"2024-09-21T21:32:15.385555+0000","last_peered":"2024-09-21T21:32:15.385555+0000","last_clean":"2024-09-21T21:32:15.385555+0000","last_became_active":"2024-09-21T21:32:13.567136+0000","last_became_peered":"2024-09-21T21:32:13.567136+0000","last_unstale":"2024-09-21T21:32:15.385555+0000","last_undegraded":"2024-09-21T21:32:15.385555+0000","last_fullsized":"2024-09-21T21:32:15.385555+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.419264+0000","last_change":"2024-09-21T21:32:15.423760+0000","last_active":"2024-09-21T21:32:15.419264+0000","last_peered":"2024-09-21T21:32:15.419264+0000","last_clean":"2024-09-21T21:32:15.419264+0000","last_became_active":"2024-09-21T21:32:13.554062+0000","last_became_peered":"2024-09-21T21:32:13.554062+0000","last_unstale":"2024-09-21T21:32:15.419264+0000","last_undegraded":"2024-09-21T21:32:15.419264+0000","last_fullsized":"2024-09-21T21:32:15.419264+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.388800+0000","last_change":"2024-09-21T21:32:15.391423+0000","last_active":"2024-09-21T21:32:15.388800+0000","last_peered":"2024-09-21T21:32:15.388800+0000","last_clean":"2024-09-21T21:32:15.388800+0000","last_became_active":"2024-09-21T21:32:13.527989+0000","last_became_peered":"2024-09-21T21:32:13.527989+0000","last_unstale":"2024-09-21T21:32:15.388800+0000","last_undegraded":"2024-09-21T21:32:15.388800+0000","last_fullsized":"2024-09-21T21:32:15.388800+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.423219+0000","last_change":"2024-09-21T21:32:13.561135+0000","last_active":"2024-09-21T21:32:15.423219+0000","last_peered":"2024-09-21T21:32:15.423219+0000","last_clean":"2024-09-21T21:32:15.423219+0000","last_became_active":"2024-09-21T21:32:13.521954+0000","last_became_peered":"2024-09-21T21:32:13.521954+0000","last_unstale":"2024-09-21T21:32:15.423219+0000","last_undegraded":"2024-09-21T21:32:15.423219+0000","last_fullsized":"2024-09-21T21:32:15.423219+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.389007+0000","last_change":"2024-09-21T21:32:15.395999+0000","last_active":"2024-09-21T21:32:15.389007+0000","last_peered":"2024-09-21T21:32:15.389007+0000","last_clean":"2024-09-21T21:32:15.389007+0000","last_became_active":"2024-09-21T21:32:13.585523+0000","last_became_peered":"2024-09-21T21:32:13.585523+0000","last_unstale":"2024-09-21T21:32:15.389007+0000","last_undegraded":"2024-09-21T21:32:15.389007+0000","last_fullsized":"2024-09-21T21:32:15.389007+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.401388+0000","last_change":"2024-09-21T21:32:15.401388+0000","last_active":"2024-09-21T21:32:15.401388+0000","last_peered":"2024-09-21T21:32:15.401388+0000","last_clean":"2024-09-21T21:32:15.401388+0000","last_became_active":"2024-09-21T21:32:13.477120+0000","last_became_peered":"2024-09-21T21:32:13.477120+0000","last_unstale":"2024-09-21T21:32:15.401388+0000","last_undegraded":"2024-09-21T21:32:15.401388+0000","last_fullsized":"2024-09-21T21:32:15.401388+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.419106+0000","last_change":"2024-09-21T21:32:15.423943+0000","last_active":"2024-09-21T21:32:15.419106+0000","last_peered":"2024-09-21T21:32:15.419106+0000","last_clean":"2024-09-21T21:32:15.419106+0000","last_became_active":"2024-09-21T21:32:13.633722+0000","last_became_peered":"2024-09-21T21:32:13.633722+0000","last_unstale":"2024-09-21T21:32:15.419106+0000","last_undegraded":"2024-09-21T21:32:15.419106+0000","last_fullsized":"2024-09-21T21:32:15.419106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.386747+0000","last_change":"2024-09-21T21:32:15.391412+0000","last_active":"2024-09-21T21:32:15.386747+0000","last_peered":"2024-09-21T21:32:15.386747+0000","last_clean":"2024-09-21T21:32:15.386747+0000","last_became_active":"2024-09-21T21:32:13.587113+0000","last_became_peered":"2024-09-21T21:32:13.587113+0000","last_unstale":"2024-09-21T21:32:15.386747+0000","last_undegraded":"2024-09-21T21:32:15.386747+0000","last_fullsized":"2024-09-21T21:32:15.386747+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:32:23.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:32:23.287 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:23.282+0000 7f54040fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:23.289 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:23.288+0000 7f54040fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:23.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:23.484 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-09-21T21:32:23.497 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-21T21:32:22.270312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.786621"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.389269+0000","last_change":"2024-09-21T21:32:15.395284+0000","last_active":"2024-09-21T21:32:15.389269+0000","last_peered":"2024-09-21T21:32:15.389269+0000","last_clean":"2024-09-21T21:32:15.389269+0000","last_became_active":"2024-09-21T21:32:13.513751+0000","last_became_peered":"2024-09-21T21:32:13.513751+0000","last_unstale":"2024-09-21T21:32:15.389269+0000","last_undegraded":"2024-09-21T21:32:15.389269+0000","last_fullsized":"2024-09-21T21:32:15.389269+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.385555+0000","last_change":"2024-09-21T21:32:15.391507+0000","last_active":"2024-09-21T21:32:15.385555+0000","last_peered":"2024-09-21T21:32:15.385555+0000","last_clean":"2024-09-21T21:32:15.385555+0000","last_became_active":"2024-09-21T21:32:13.567136+0000","last_became_peered":"2024-09-21T21:32:13.567136+0000","last_unstale":"2024-09-21T21:32:15.385555+0000","last_undegraded":"2024-09-21T21:32:15.385555+0000","last_fullsized":"2024-09-21T21:32:15.385555+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.419264+0000","last_change":"2024-09-21T21:32:15.423760+0000","last_active":"2024-09-21T21:32:15.419264+0000","last_peered":"2024-09-21T21:32:15.419264+0000","last_clean":"2024-09-21T21:32:15.419264+0000","last_became_active":"2024-09-21T21:32:13.554062+0000","last_became_peered":"2024-09-21T21:32:13.554062+0000","last_unstale":"2024-09-21T21:32:15.419264+0000","last_undegraded":"2024-09-21T21:32:15.419264+0000","last_fullsized":"2024-09-21T21:32:15.419264+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.388800+0000","last_change":"2024-09-21T21:32:15.391423+0000","last_active":"2024-09-21T21:32:15.388800+0000","last_peered":"2024-09-21T21:32:15.388800+0000","last_clean":"2024-09-21T21:32:15.388800+0000","last_became_active":"2024-09-21T21:32:13.527989+0000","last_became_peered":"2024-09-21T21:32:13.527989+0000","last_unstale":"2024-09-21T21:32:15.388800+0000","last_undegraded":"2024-09-21T21:32:15.388800+0000","last_fullsized":"2024-09-21T21:32:15.388800+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.423219+0000","last_change":"2024-09-21T21:32:13.561135+0000","last_active":"2024-09-21T21:32:15.423219+0000","last_peered":"2024-09-21T21:32:15.423219+0000","last_clean":"2024-09-21T21:32:15.423219+0000","last_became_active":"2024-09-21T21:32:13.521954+0000","last_became_peered":"2024-09-21T21:32:13.521954+0000","last_unstale":"2024-09-21T21:32:15.423219+0000","last_undegraded":"2024-09-21T21:32:15.423219+0000","last_fullsized":"2024-09-21T21:32:15.423219+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.389007+0000","last_change":"2024-09-21T21:32:15.395999+0000","last_active":"2024-09-21T21:32:15.389007+0000","last_peered":"2024-09-21T21:32:15.389007+0000","last_clean":"2024-09-21T21:32:15.389007+0000","last_became_active":"2024-09-21T21:32:13.585523+0000","last_became_peered":"2024-09-21T21:32:13.585523+0000","last_unstale":"2024-09-21T21:32:15.389007+0000","last_undegraded":"2024-09-21T21:32:15.389007+0000","last_fullsized":"2024-09-21T21:32:15.389007+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.401388+0000","last_change":"2024-09-21T21:32:15.401388+0000","last_active":"2024-09-21T21:32:15.401388+0000","last_peered":"2024-09-21T21:32:15.401388+0000","last_clean":"2024-09-21T21:32:15.401388+0000","last_became_active":"2024-09-21T21:32:13.477120+0000","last_became_peered":"2024-09-21T21:32:13.477120+0000","last_unstale":"2024-09-21T21:32:15.401388+0000","last_undegraded":"2024-09-21T21:32:15.401388+0000","last_fullsized":"2024-09-21T21:32:15.401388+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.419106+0000","last_change":"2024-09-21T21:32:15.423943+0000","last_active":"2024-09-21T21:32:15.419106+0000","last_peered":"2024-09-21T21:32:15.419106+0000","last_clean":"2024-09-21T21:32:15.419106+0000","last_became_active":"2024-09-21T21:32:13.633722+0000","last_became_peered":"2024-09-21T21:32:13.633722+0000","last_unstale":"2024-09-21T21:32:15.419106+0000","last_undegraded":"2024-09-21T21:32:15.419106+0000","last_fullsized":"2024-09-21T21:32:15.419106+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-21T21:32:15.386747+0000","last_change":"2024-09-21T21:32:15.391412+0000","last_active":"2024-09-21T21:32:15.386747+0000","last_peered":"2024-09-21T21:32:15.386747+0000","last_clean":"2024-09-21T21:32:15.386747+0000","last_became_active":"2024-09-21T21:32:13.587113+0000","last_became_peered":"2024-09-21T21:32:13.587113+0000","last_unstale":"2024-09-21T21:32:15.386747+0000","last_undegraded":"2024-09-21T21:32:15.386747+0000","last_fullsized":"2024-09-21T21:32:15.386747+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:32:12.346978+0000","last_clean_scrub_stamp":"2024-09-21T21:32:12.346978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:32:23.498 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T21:32:23.498 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T21:32:23.498 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T21:32:23.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T21:32:23.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:23.607+0000 7f7946ebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:23.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:32:23.611+0000 7f7946ebd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:32:23.848 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:23.849 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T21:32:23.861 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T21:32:23.862 INFO:teuthology.run_tasks:Running task cbt... 2024-09-21T21:32:23.874 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi063.front.sepia.ceph.com', 'osds': ['ubuntu@smithi063.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi063.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi063.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi063.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-21T21:32:23.875 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-21T21:32:23.893 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:32:23.893 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:32:23.946 DEBUG:teuthology.orchestra.run.smithi063:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-21T21:32:24.002 INFO:teuthology.orchestra.run.smithi063.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-21T21:32:24.309 INFO:tasks.cbt:Installing collectl 2024-09-21T21:32:24.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-21T21:32:24.417 INFO:teuthology.orchestra.run.smithi063.stderr:--2024-09-21 21:32:24-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-21T21:32:24.600 INFO:teuthology.orchestra.run.smithi063.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2024-09-21T21:32:24.609 INFO:teuthology.orchestra.run.smithi063.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-09-21T21:32:24.834 INFO:teuthology.orchestra.run.smithi063.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:32:24.834 INFO:teuthology.orchestra.run.smithi063.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zto45xR0rCNCsv2E9J2loxmBsG3yYVVnPon0SFKnH1iLL7cJWtOCeZWAVr7zk9rFgfHI26FGAWe2-tA_VR3QVONjQ%3D%3D&use_mirror=netactuate&r= [following] 2024-09-21T21:32:24.834 INFO:teuthology.orchestra.run.smithi063.stderr:--2024-09-21 21:32:24-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zto45xR0rCNCsv2E9J2loxmBsG3yYVVnPon0SFKnH1iLL7cJWtOCeZWAVr7zk9rFgfHI26FGAWe2-tA_VR3QVONjQ%3D%3D&use_mirror=netactuate&r= 2024-09-21T21:32:24.834 INFO:teuthology.orchestra.run.smithi063.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-21T21:32:24.906 INFO:teuthology.orchestra.run.smithi063.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-21T21:32:25.177 INFO:teuthology.orchestra.run.smithi063.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:32:25.177 INFO:teuthology.orchestra.run.smithi063.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-21T21:32:25.178 INFO:teuthology.orchestra.run.smithi063.stderr:--2024-09-21 21:32:25-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-21T21:32:25.188 INFO:teuthology.orchestra.run.smithi063.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2024-09-21T21:32:25.209 INFO:teuthology.orchestra.run.smithi063.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2024-09-21T21:32:25.335 INFO:teuthology.orchestra.run.smithi063.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-21T21:32:25.335 INFO:teuthology.orchestra.run.smithi063.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-21T21:32:25.335 INFO:teuthology.orchestra.run.smithi063.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-21T21:32:25.335 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:32:25.375 INFO:teuthology.orchestra.run.smithi063.stderr: 0K .......... .......... .......... .......... .......... 8% 1.21M 0s 2024-09-21T21:32:25.396 INFO:teuthology.orchestra.run.smithi063.stderr: 50K .......... .......... .......... .......... .......... 16% 2.40M 0s 2024-09-21T21:32:25.418 INFO:teuthology.orchestra.run.smithi063.stderr: 100K .......... .......... .......... .......... .......... 24% 2.41M 0s 2024-09-21T21:32:25.418 INFO:teuthology.orchestra.run.smithi063.stderr: 150K .......... .......... .......... .......... .......... 32% 121M 0s 2024-09-21T21:32:25.440 INFO:teuthology.orchestra.run.smithi063.stderr: 200K .......... .......... .......... .......... .......... 41% 2.48M 0s 2024-09-21T21:32:25.441 INFO:teuthology.orchestra.run.smithi063.stderr: 250K .......... .......... .......... .......... .......... 49% 119M 0s 2024-09-21T21:32:25.441 INFO:teuthology.orchestra.run.smithi063.stderr: 300K .......... .......... .......... .......... .......... 57% 115M 0s 2024-09-21T21:32:25.441 INFO:teuthology.orchestra.run.smithi063.stderr: 350K .......... .......... .......... .......... .......... 65% 93.7M 0s 2024-09-21T21:32:25.441 INFO:teuthology.orchestra.run.smithi063.stderr: 400K .......... .......... .......... .......... .......... 74% 114M 0s 2024-09-21T21:32:25.458 INFO:teuthology.orchestra.run.smithi063.stderr: 450K .......... .......... .......... .......... .......... 82% 2.62M 0s 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr: 500K .......... .......... .......... .......... .......... 90% 124M 0s 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr: 550K .......... .......... .......... .......... .......... 98% 101M 0s 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr: 600K ...... 100% 11.4T=0.1s 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21 21:32:25 (4.83 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-21T21:32:25.459 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/ 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/collectl.conf 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/ 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Socket.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/colmux.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-21T21:32:25.493 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Import.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/style.css 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Process.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-21T21:32:25.494 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Export.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Startup.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Performance.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Messages.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Logging.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-21T21:32:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Misc.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Examples.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Data.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-21T21:32:25.496 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Playback.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Hello.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Network.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Memory.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/Disks.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/service/ 2024-09-21T21:32:25.497 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/service/collectl.service 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/initd/ 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/initd/collectl 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/misc.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/gexpr.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/statsd.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/envrules.std 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/hello.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/vmsum.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/proctree.ph 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/README-WINDOWS 2024-09-21T21:32:25.498 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/formatit.ph 2024-09-21T21:32:25.499 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/colmux 2024-09-21T21:32:25.499 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/INSTALL 2024-09-21T21:32:25.499 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/graphite.ph 2024-09-21T21:32:25.499 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/vmstat.ph 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/README 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/man1/ 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/man1/collectl.1 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/man1/colmux.1 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/lexpr.ph 2024-09-21T21:32:25.500 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/client.pl 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/collectl 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/COPYING 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/vnet.ph 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/GPL 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/UNINSTALL 2024-09-21T21:32:25.501 INFO:teuthology.orchestra.run.smithi063.stdout:collectl-4.3.1/ARTISTIC 2024-09-21T21:32:25.553 INFO:teuthology.orchestra.run.smithi063.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-21T21:32:25.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-21T21:32:26.173 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:03:05 ago on Sat 21 Sep 2024 09:29:21 PM UTC. 2024-09-21T21:32:26.268 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-21T21:32:26.271 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-21T21:32:26.323 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-09-21T21:32:26.323 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-09-21T21:32:26.324 INFO:teuthology.orchestra.run.smithi063.stdout:Install 5 Packages 2024-09-21T21:32:26.325 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:26.325 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.4 M 2024-09-21T21:32:26.325 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 11 M 2024-09-21T21:32:26.325 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-09-21T21:32:26.794 INFO:teuthology.orchestra.run.smithi063.stdout:(1/5): libradospp-devel-19.3.0-5092.g62da54b4.e 192 kB/s | 31 kB 00:00 2024-09-21T21:32:26.819 INFO:teuthology.orchestra.run.smithi063.stdout:(2/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 128 kB/s | 24 kB 00:00 2024-09-21T21:32:26.861 INFO:teuthology.orchestra.run.smithi063.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 315 kB/s | 13 kB 00:00 2024-09-21T21:32:26.886 INFO:teuthology.orchestra.run.smithi063.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 1.3 MB/s | 118 kB 00:00 2024-09-21T21:32:26.937 INFO:teuthology.orchestra.run.smithi063.stdout:(5/5): perf-5.14.0-508.el9.x86_64.rpm 7.4 MB/s | 2.2 MB 00:00 2024-09-21T21:32:26.939 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:32:26.939 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.9 MB/s | 2.4 MB 00:00 2024-09-21T21:32:26.970 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-09-21T21:32:26.992 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-09-21T21:32:26.992 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-09-21T21:32:27.126 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-09-21T21:32:27.127 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-09-21T21:32:27.490 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-09-21T21:32:27.556 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-21T21:32:27.595 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-21T21:32:27.625 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-21T21:32:27.976 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-21T21:32:28.058 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:32:29.716 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:32:29.716 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-21T21:32:29.716 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-21T21:32:29.716 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-21T21:32:29.717 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-21T21:32:30.081 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: pdsh-2.34-7.el9.x86_64 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-21T21:32:30.082 INFO:teuthology.orchestra.run.smithi063.stdout: perf-5.14.0-508.el9.x86_64 2024-09-21T21:32:30.083 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-09-21T21:32:30.083 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-09-21T21:32:30.433 INFO:tasks.cbt:benchmark: radosbench 2024-09-21T21:32:30.434 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-21T21:32:30.460 DEBUG:teuthology.orchestra.run.smithi063:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - Settings.cluster: 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'clients': ['ubuntu@smithi063.front.sepia.ceph.com'], 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'head': 'ubuntu@smithi063.front.sepia.ceph.com', 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'is_teuthology': True, 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'iterations': 1, 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'mons': ['ubuntu@smithi063.front.sepia.ceph.com'], 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'osds': ['ubuntu@smithi063.front.sepia.ceph.com'], 2024-09-21T21:32:30.863 INFO:teuthology.orchestra.run.smithi063.stderr: 'osds_per_node': 3, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'pid_dir': '/var/run/ceph', 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'pgp_size': 256, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'replication': 'replicated'}}, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'rebuild_every_test': False, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'rgws': {'ubuntu@smithi063.front.sepia.ceph.com': None}, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'use_existing': True, 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr: 'user': 'ubuntu'} 2024-09-21T21:32:30.864 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2024-09-21T21:32:30.884 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-21T21:32:30.885 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-09-21T21:32:30.888 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:32:30.890 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-09-21T21:32:30.893 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:32:30.893 INFO:teuthology.orchestra.run.smithi063.stderr:21:32:30 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:33:30.955 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:33:31.008 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:33:31.050 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-09-21T21:33:31.052 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-09-21T21:33:31.054 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-09-21T21:33:31.069 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/$(basename ${i}).smithi063.front.sepia.ceph.com ; done 2024-09-21T21:33:31.074 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-09-21T21:33:31.096 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - WARNING - cbt - sudo killall -9 rados 2024-09-21T21:33:31.096 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - WARNING - cbt - error 1 seen, continuing anyway... 2024-09-21T21:33:31.097 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2024-09-21T21:33:31.116 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c. 2024-09-21T21:33:31.116 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-21T21:33:31.116 INFO:teuthology.orchestra.run.smithi063.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c 2024-09-21T21:33:31.117 INFO:teuthology.orchestra.run.smithi063.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'readmode': 'rand', 'time': 60, 'write_only': False, 'op_size': 4194304, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2024-09-21T21:33:31.117 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-21T21:33:31.117 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-09-21T21:33:31.131 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-09-21T21:33:31.133 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:33:31.135 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-09-21T21:33:31.138 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:33:31.138 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:33:31.138 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-09-21T21:33:31.457 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-09-21T21:33:32.456 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-09-21T21:33:33.458 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:33 - INFO - cbt - Checking Health after pool creation. 2024-09-21T21:33:38.463 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:33:39.039 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:39 - INFO - cbt - Final Pool Health Check. 2024-09-21T21:33:44.044 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:33:44.394 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:33:44.418 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:33:44.455 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-09-21T21:33:44.457 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-09-21T21:33:44.459 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-21T21:33:44.539 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:44 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-21T21:33:45.090 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-21T21:33:45.397 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-09-21T21:33:45.468 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-09-21T21:33:46.281 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-09-21T21:33:46.283 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:33:46.285 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-09-21T21:33:46.290 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:33:46.290 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:33:46.290 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - INFO - cbt - Running radosbench write test. 2024-09-21T21:33:46.290 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-09-21T21:33:46.290 INFO:teuthology.orchestra.run.smithi063.stderr:21:33:46 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-09-21T21:34:49.437 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:34:49.472 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:34:49.507 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-09-21T21:34:49.508 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-09-21T21:34:49.510 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-09-21T21:34:49.870 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-09-21T21:34:49.885 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/$(basename ${i}).smithi063.front.sepia.ceph.com ; done 2024-09-21T21:34:49.922 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - INFO - cbt - Convert results to json format. 2024-09-21T21:34:49.923 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-21T21:34:50.036 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-21T21:34:50.265 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-21T21:34:50.599 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand 2024-09-21T21:34:50.653 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:50 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/ceph_settings.out 2024-09-21T21:34:51.472 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl 2024-09-21T21:34:51.475 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:34:51.477 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf 2024-09-21T21:34:51.480 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:34:51.481 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:34:51.481 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - INFO - cbt - Running radosbench rand test. 2024-09-21T21:34:51.481 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.0 2024-09-21T21:34:51.482 INFO:teuthology.orchestra.run.smithi063.stderr:21:34:51 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/output.1 2024-09-21T21:35:52.637 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:35:52.894 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:35:52.929 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf.data 2024-09-21T21:35:52.931 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/perf/perf_stat.* 2024-09-21T21:35:52.932 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/historic_ops.out 2024-09-21T21:35:53.344 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* 2024-09-21T21:35:53.359 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/$(basename ${i}).smithi063.front.sepia.ceph.com ; done 2024-09-21T21:35:53.533 INFO:teuthology.orchestra.run.smithi063.stderr:21:35:53 - INFO - cbt - Convert results to json format. 2024-09-21T21:35:53.705 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-09-21T21:35:53.714 DEBUG:teuthology.orchestra.run.smithi063:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-09-21T21:35:53.940 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-09-21T21:35:53.940 INFO:tasks.cbt_performance:reading cbt preferences 2024-09-21T21:35:53.941 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-09-21T21:35:54.020 INFO:teuthology.orchestra.run.smithi063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi063.front.sepia.ceph.com 2024-09-21T21:35:54.020 INFO:teuthology.orchestra.run.smithi063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi063.front.sepia.ceph.com 2024-09-21T21:35:54.020 INFO:teuthology.orchestra.run.smithi063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi063.front.sepia.ceph.com 2024-09-21T21:35:54.020 INFO:teuthology.orchestra.run.smithi063.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi063.front.sepia.ceph.com 2024-09-21T21:35:54.021 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:35:54.021 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.1.smithi063.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:35:54.041 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:35:54.041 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-09-21T21:35:54.097 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:35:54.097 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:35:54.097 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:35:54.097 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand/json_output.0.smithi063.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:35:54.154 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:35:54.155 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/rand -name 'perf_stat.*' 2024-09-21T21:35:54.211 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:35:54.211 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:35:54.211 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:35:54.211 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.1.smithi063.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:35:54.323 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:35:54.323 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-09-21T21:35:54.380 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:35:54.380 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:35:54.380 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-09-21T21:35:54.380 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write/json_output.0.smithi063.front.sepia.ceph.com of=/dev/stdout 2024-09-21T21:35:54.448 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-21T21:35:54.448 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3b12466c/write -name 'perf_stat.*' 2024-09-21T21:35:54.504 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-21T21:35:54.504 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-21T21:35:54.512 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:35:54.518 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:35:54.524 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:35:54.530 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-21T21:35:54.530 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T21:35:54.539 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T21:35:54.675 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:35:54.667+0000 7fddc2351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:35:54.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-09-21T21:35:54.676+0000 7fddc2351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:35:54.904 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T21:35:54.904 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T21:35:54.904 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T21:35:54.904 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:35:54.993 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T21:35:54.994 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T21:35:54.994 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:35:55.089 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T21:35:55.089 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T21:35:55.089 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:35:55.182 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T21:35:55.182 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T21:35:55.183 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T21:35:55.183 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:35:55.213 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T21:35:55.213 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T21:35:55.213 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T21:35:55.213 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T21:35:55.264 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T21:35:55.264 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T21:35:55.264 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T21:35:55.359 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:35:55.360 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T21:35:55.651 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:35:55.651 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T21:35:55.701 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:35:55.701 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T21:35:55.742 INFO:tasks.ceph:Archiving mon data... 2024-09-21T21:35:55.743 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352/data/mon.a.tgz 2024-09-21T21:35:55.743 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-09-21T21:35:55.809 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.7WtvUjcMMx 2024-09-21T21:35:55.809 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7WtvUjcMMx 2024-09-21T21:35:55.929 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.7WtvUjcMMx 2024-09-21T21:35:56.059 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.7WtvUjcMMx is 212KB 2024-09-21T21:35:56.151 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.7WtvUjcMMx 2024-09-21T21:35:56.166 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T21:35:56.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T21:35:56.287 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T21:35:56.287 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T21:35:56.289 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352/remote/smithi063/crash 2024-09-21T21:35:56.289 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T21:35:56.352 INFO:tasks.ceph:Compressing logs... 2024-09-21T21:35:56.352 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T21:35:56.435 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T21:35:56.435 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31086.log 2024-09-21T21:35:56.436 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31308.log 2024-09-21T21:35:56.436 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.31086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29808.log 2024-09-21T21:35:56.436 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.31308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29555.log 2024-09-21T21:35:56.436 INFO:teuthology.orchestra.run.smithi063.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.31308.log.gz 2024-09-21T21:35:56.436 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30167.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29683.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29433.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29196.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28855.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31002.log 2024-09-21T21:35:56.437 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27619.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29895.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29349.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29050.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31608.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2024-09-21T21:35:56.439 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31151.log 2024-09-21T21:35:56.440 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29809.log 2024-09-21T21:35:56.441 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29935.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30273.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30315.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31307.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29020.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31039.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29475.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28573.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30147.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30021.log 2024-09-21T21:35:56.442 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31150.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29391.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30062.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28771.log 2024-09-21T21:35:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28811.log 2024-09-21T21:35:56.445 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T21:35:56.445 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31063.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29279.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30041.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29769.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29556.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29934.log 2024-09-21T21:35:56.446 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29557.log 2024-09-21T21:35:56.447 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28661.log 2024-09-21T21:35:56.447 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30937.log 2024-09-21T21:35:56.447 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29682.log 2024-09-21T21:35:56.448 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29252.log 2024-09-21T21:35:56.459 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29555.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.28661.log: /var/log/ceph/ceph-client.admin.31608.log: /var/log/ceph/ceph-client.admin.29682.log: /var/log/ceph/ceph-client.admin.29643.log: /var/log/ceph/ceph-client.admin.29556.log: /var/log/ceph/ceph-client.admin.29557.log: /var/log/ceph/ceph-client.admin.30937.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.31307.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31608.log.gz 2024-09-21T21:35:56.459 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-mgr.x.log: 66.1% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31307.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31086.log.gz 2024-09-21T21:35:56.460 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:35:56.460 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29808.log: /var/log/ceph/ceph-client.admin.29683.log: /var/log/ceph/ceph-client.admin.29895.log: /var/log/ceph/ceph-client.admin.29433.log: /var/log/ceph/ceph-client.admin.29196.log: /var/log/ceph/ceph-client.admin.30167.log: /var/log/ceph/ceph-client.admin.28855.log: /var/log/ceph/ceph-client.admin.29050.log: /var/log/ceph/ceph-client.admin.29252.log: /var/log/ceph/ceph-client.admin.29935.log: /var/log/ceph/ceph-client.admin.30315.log: /var/log/ceph/ceph-client.admin.31002.log: /var/log/ceph/ceph-client.admin.31151.log: /var/log/ceph/ceph.tmp-client.admin.27619.log: /var/log/ceph/ceph-client.admin.29475.log: /var/log/ceph/ceph-client.admin.29020.log: /var/log/ceph/ceph-client.admin.30231.log: /var/log/ceph/ceph-client.admin.29349.log: /var/log/ceph/ceph-client.admin.29809.log: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T21:35:56.472 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/ceph-client.admin.29391.log: /var/log/ceph/ceph-client.admin.30147.log: /var/log/ceph/ceph-client.admin.30273.log: /var/log/ceph/ceph-client.admin.31039.log: /var/log/ceph/ceph-client.admin.28573.log: /var/log/ceph/ceph-client.admin.30021.log: /var/log/ceph/ceph-client.admin.30062.log: /var/log/ceph/ceph-client.admin.28811.log: /var/log/ceph/ceph-client.admin.31150.log: /var/log/ceph/ceph-client.admin.30041.log: /var/log/ceph/ceph-client.admin.28771.log: /var/log/ceph/ceph-client.admin.29769.log: /var/log/ceph/ceph-client.admin.29934.log: /var/log/ceph/ceph-client.admin.29279.log: /var/log/ceph/ceph-client.admin.31063.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31002.log.gz 2024-09-21T21:35:56.473 INFO:teuthology.orchestra.run.smithi063.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.31151.log.gz 2024-09-21T21:35:56.474 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28661.log.gz 2024-09-21T21:35:56.474 INFO:teuthology.orchestra.run.smithi063.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.31039.log.gz 2024-09-21T21:35:56.474 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31063.log.gz 2024-09-21T21:35:56.474 INFO:teuthology.orchestra.run.smithi063.stderr: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.31150.log.gz 2024-09-21T21:35:56.477 INFO:teuthology.orchestra.run.smithi063.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29020.log.gz 50.0% 2024-09-21T21:35:56.477 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29050.log.gz 2024-09-21T21:35:56.477 INFO:teuthology.orchestra.run.smithi063.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T21:35:56.478 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28573.log.gz 2024-09-21T21:35:56.480 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28855.log.gz 2024-09-21T21:35:56.482 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30937.log.gz 2024-09-21T21:35:56.498 INFO:teuthology.orchestra.run.smithi063.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27619.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29196.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29252.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29349.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29391.log.gz 66.9% 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29279.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29433.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29475.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29556.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29557.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29555.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29683.log.gz 2024-09-21T21:35:56.505 INFO:teuthology.orchestra.run.smithi063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29682.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29769.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29808.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29809.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29895.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29935.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30021.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29934.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30147.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30041.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30062.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30167.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30315.log.gz 2024-09-21T21:35:56.506 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30273.log.gz 2024-09-21T21:35:56.520 INFO:teuthology.orchestra.run.smithi063.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T21:35:56.521 INFO:teuthology.orchestra.run.smithi063.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28771.log.gz 2024-09-21T21:35:56.522 INFO:teuthology.orchestra.run.smithi063.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28811.log.gz 2024-09-21T21:35:56.588 INFO:teuthology.orchestra.run.smithi063.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T21:36:03.923 INFO:teuthology.orchestra.run.smithi063.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T21:36:04.830 INFO:teuthology.orchestra.run.smithi063.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T21:36:06.456 INFO:teuthology.orchestra.run.smithi063.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T21:36:06.459 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-09-21T21:36:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:real 0m10.051s 2024-09-21T21:36:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:user 0m6.818s 2024-09-21T21:36:06.459 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m0.343s 2024-09-21T21:36:06.459 INFO:tasks.ceph:Archiving logs... 2024-09-21T21:36:06.459 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352/remote/smithi063/log 2024-09-21T21:36:06.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T21:36:07.546 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T21:36:07.555 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T21:36:07.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T21:36:07.616 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T21:36:07.624 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T21:36:07.625 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:36:07.649 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +258us[ +259us] +/- 41ms 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -897us[ -897us] +/- 58ms 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +606us[ +606us] +/- 75ms 2024-09-21T21:36:07.729 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:36:07.742 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T21:36:07.751 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T21:36:07.751 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T21:36:07.766 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T21:36:07.908 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:36:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:36:07.937 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T21:36:07.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T21:36:07.972 INFO:teuthology.task.internal:Duration was 652.494792 seconds 2024-09-21T21:36:07.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T21:36:07.980 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T21:36:07.981 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T21:36:08.122 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:36:08.688 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T21:36:08.688 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-09-21T21:36:08.688 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T21:36:08.711 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T21:36:08.711 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T21:36:08.780 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T21:36:08.781 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T21:36:08.937 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T21:36:08.949 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T21:36:08.949 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T21:36:08.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T21:36:09.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:36:09.068 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-09-21T21:36:09.097 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T21:36:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:36:09.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T21:36:09.163 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T21:36:09.164 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914352/remote/smithi063 2024-09-21T21:36:09.164 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T21:36:09.298 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T21:36:09.298 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T21:36:09.333 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T21:36:09.351 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T21:36:09.352 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T21:36:09.381 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T21:36:09.382 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T21:36:09.397 INFO:teuthology.orchestra.run.smithi063.stdout: 265958 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 21:36 /home/ubuntu/cephtest 2024-09-21T21:36:09.398 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T21:36:09.408 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T21:36:09.430 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7914352 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/radosbench_4M_rand_read} duration: 652.4947917461395 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-21T21:36:09.430 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:36:09.512 INFO:teuthology.run:pass