2024-09-21T21:10:19.843 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T21:10:19.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:10:19.948 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914337 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914337' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0FTx0oLkzEd2LoLzZYPS3ETIjYjRH/HIlJ7/+EWbU+QuO8JYLOd4vTGAzNJgxCrfW/DVKYcXXo7mKefWuybBA= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T21:10:19.949 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T21:10:19.949 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T21:10:19.950 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T21:10:19.951 INFO:teuthology.task.internal:Checking packages... 2024-09-21T21:10:19.971 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T21:10:19.971 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T21:10:19.971 INFO:teuthology.packaging:ref: None 2024-09-21T21:10:19.971 INFO:teuthology.packaging:tag: None 2024-09-21T21:10:19.972 INFO:teuthology.packaging:branch: main 2024-09-21T21:10:19.972 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:10:19.972 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T21:10:20.196 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T21:10:20.197 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T21:10:20.206 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T21:10:20.206 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T21:10:20.211 INFO:teuthology.task.internal:Saving configuration 2024-09-21T21:10:20.227 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T21:10:20.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T21:10:20.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914337', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:05:07.771040', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0FTx0oLkzEd2LoLzZYPS3ETIjYjRH/HIlJ7/+EWbU+QuO8JYLOd4vTGAzNJgxCrfW/DVKYcXXo7mKefWuybBA='} 2024-09-21T21:10:20.271 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T21:10:20.276 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-21T21:10:20.276 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T21:10:20.388 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f38cc020160>, signals=[15]) 2024-09-21T21:10:20.388 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T21:10:20.394 INFO:teuthology.task.internal:Opening connections... 2024-09-21T21:10:20.394 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T21:10:20.395 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:10:20.470 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T21:10:20.475 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-09-21T21:10:20.556 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-09-21T21:10:20.556 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-09-21T21:10:20.614 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-09-21T21:10:20.614 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T21:10:20.615 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T21:10:20.616 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-09-21T21:10:20.639 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T21:10:20.647 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T21:10:20.654 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T21:10:20.654 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:10:20.671 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T21:10:20.676 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T21:10:20.676 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:10:20.743 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T21:10:20.772 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T21:10:20.785 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T21:10:20.785 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T21:10:20.785 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:10:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:10:20.802 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-21T21:10:20.859 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-503.el9.x86_64 2024-09-21T21:10:20.859 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-503.el9.x86_64 2024-09-21T21:10:20.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-09-21T21:10:23.466 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.3 MB 00:00 2024-09-21T21:10:26.842 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 8.8 MB/s | 20 MB 00:02 2024-09-21T21:10:30.984 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-21T21:10:32.651 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 19 kB 00:00 2024-09-21T21:10:33.655 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-21T21:10:39.202 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-09-21T21:10:41.108 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T21:10:41.109 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T21:10:41.192 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:10:41.198 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 79 M 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 126 M 2024-09-21T21:10:41.199 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:10:41.420 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 390 kB/s | 36 kB 00:00 2024-09-21T21:10:41.929 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-21T21:10:42.287 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 36 MB/s | 35 MB 00:00 2024-09-21T21:10:42.571 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-21T21:10:42.571 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:10:42.571 INFO:teuthology.orchestra.run.smithi107.stdout:Total 57 MB/s | 79 MB 00:01 2024-09-21T21:10:43.257 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:10:43.604 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:10:43.604 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:10:44.554 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:10:44.554 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:10:46.983 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:10:47.707 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:10:47.813 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:10:50.587 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:10:50.715 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:10:55.575 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:10:55.623 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:00.484 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:38.769 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:38.809 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T21:11:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T21:11:39.087 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T21:11:39.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T21:11:39.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:11:39.944 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:11:40.296 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:11:40.380 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:11:40.380 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:11:40.380 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T21:11:40.381 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:11:40.381 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:11:40.381 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T21:11:40.381 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:11:40.381 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T21:11:40.439 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T21:11:40.439 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T21:11:40.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 82.el9 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8274670 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T21:11:40.471 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:11:40.472 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T21:11:40.473 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T21:11:40.473 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-508.el9.x86_64 2024-09-21T21:11:40.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T21:11:41.181 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-09-21T21:11:42.522 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T21:11:42.551 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-09-21T21:11:42.553 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-09-21T21:11:42.569 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.vMTuwijg9B 2024-09-21T21:11:42.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vMTuwijg9B 2024-09-21T21:11:42.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.vMTuwijg9B 2024-09-21T21:11:42.783 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.vMTuwijg9B is 6KB 2024-09-21T21:11:42.831 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.vMTuwijg9B 2024-09-21T21:11:42.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T21:11:42.910 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T21:11:42.910 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T21:11:42.910 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T21:11:42.910 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T21:11:42.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T21:11:43.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-09-21T21:12:13.097 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T21:12:13.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:12:13.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:12:31.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-21T21:12:40.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:12:40.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:12:43.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-21T21:12:55.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:12:55.537 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:13:13.967 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-21T21:13:28.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:13:28.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:13:29.361 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-21T21:13:29.804 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:13:29.805 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T21:13:29.805 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-21T21:13:29.819 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-508.el9.x86_64 2024-09-21T21:13:29.819 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T21:13:29.820 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:13:29.820 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T21:13:30.821 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T21:13:30.821 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:13:30.891 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-09-21T21:13:30.916 DEBUG:teuthology.parallel:result is None 2024-09-21T21:13:30.916 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T21:13:30.923 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T21:13:30.923 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:13:30.939 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T21:13:30.945 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T21:13:30.951 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T21:13:30.951 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:13:31.002 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T21:13:31.009 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T21:13:31.010 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:13:31.079 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:13:31.090 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:13:31.091 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T21:13:31.098 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T21:13:31.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:13:31.159 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T21:13:31.167 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T21:13:31.168 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:13:31.219 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:13:31.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:13:31.361 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:13:31.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:13:31.491 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:13:31.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:13:31.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-09-21T21:13:31.654 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T21:13:31.964 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T21:13:31.972 INFO:teuthology.task.internal:Starting timer... 2024-09-21T21:13:31.972 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T21:13:31.982 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T21:13:31.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-09-21T21:13:32.067 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-09-21T21:13:32.069 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T21:13:32.069 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-09-21T21:13:32.102 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-09-21T21:13:32.103 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2024-09-21T21:13:32.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T21:13:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:13:32.173 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T21:13:32.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-09-21T21:13:32.265 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T21:13:32.280 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T21:13:32.290 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T21:13:32.291 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T21:17:12.843 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2024-09-21T21:17:12.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:17:12.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:17:12.919 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-21T21:17:12.997 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-21T21:17:12.997 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T21:17:13.008 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T21:17:13.008 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T21:17:13.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T21:17:13.065 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T21:17:13.079 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T21:17:13.103 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-09-21T21:17:13.114 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-09-21T21:17:13.128 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T21:17:13.143 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T21:17:13.196 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:17:13.201 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T21:17:13.201 INFO:teuthology.run_tasks:Running task install... 2024-09-21T21:17:13.209 DEBUG:teuthology.task.install:project ceph 2024-09-21T21:17:13.209 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T21:17:13.209 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T21:17:13.209 INFO:teuthology.task.install:Using flavor: crimson 2024-09-21T21:17:13.214 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T21:17:13.214 INFO:teuthology.task.install:extra packages: [] 2024-09-21T21:17:13.215 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T21:17:13.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:17:13.369 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-21T21:17:13.369 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-21T21:17:13.476 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T21:17:13.477 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:17:13.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T21:17:13.503 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T21:17:13.503 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T21:17:13.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T21:17:13.642 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T21:17:13.707 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-09-21T21:17:13.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-09-21T21:17:13.937 INFO:teuthology.orchestra.run.smithi107.stdout:45 files removed 2024-09-21T21:17:13.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-09-21T21:17:14.516 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 327 kB/s | 88 kB 00:00 2024-09-21T21:17:14.808 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-21T21:17:15.083 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 8.7 kB/s | 2.1 kB 00:00 2024-09-21T21:17:15.758 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-21T21:17:17.610 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-21T21:17:22.019 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-21T21:17:23.552 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 107 kB/s | 19 kB 00:00 2024-09-21T21:17:24.998 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-21T21:17:30.412 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T21:17:32.671 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-09-21T21:17:32.684 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T21:17:32.685 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-21T21:17:32.686 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:Install 31 Packages 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-09-21T21:17:32.687 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:17:32.688 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 60 M 2024-09-21T21:17:32.688 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:17:34.721 INFO:teuthology.orchestra.run.smithi107.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 3.5 MB/s | 6.1 MB 00:01 2024-09-21T21:17:34.763 INFO:teuthology.orchestra.run.smithi107.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 609 kB/s | 25 kB 00:00 2024-09-21T21:17:34.863 INFO:teuthology.orchestra.run.smithi107.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 7.9 MB/s | 808 kB 00:00 2024-09-21T21:17:34.930 INFO:teuthology.orchestra.run.smithi107.stdout:(4/33): libradosstriper1-19.3.0-5092.g62da54b4. 8.2 MB/s | 558 kB 00:00 2024-09-21T21:17:35.205 INFO:teuthology.orchestra.run.smithi107.stdout:(5/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_6 19 MB/s | 5.4 MB 00:00 2024-09-21T21:17:35.247 INFO:teuthology.orchestra.run.smithi107.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 1.1 MB/s | 45 kB 00:00 2024-09-21T21:17:35.297 INFO:teuthology.orchestra.run.smithi107.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 2.9 MB/s | 148 kB 00:00 2024-09-21T21:17:35.348 INFO:teuthology.orchestra.run.smithi107.stdout:(8/33): python3-cephfs-19.3.0-5092.g62da54b4.el 3.1 MB/s | 159 kB 00:00 2024-09-21T21:17:35.399 INFO:teuthology.orchestra.run.smithi107.stdout:(9/33): python3-rados-19.3.0-5092.g62da54b4.el9 6.1 MB/s | 314 kB 00:00 2024-09-21T21:17:35.449 INFO:teuthology.orchestra.run.smithi107.stdout:(10/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.9 MB/s | 302 kB 00:00 2024-09-21T21:17:35.499 INFO:teuthology.orchestra.run.smithi107.stdout:(11/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 2.0 MB/s | 99 kB 00:00 2024-09-21T21:17:35.924 INFO:teuthology.orchestra.run.smithi107.stdout:(12/33): boost-program-options-1.75.0-8.el9.x86 252 kB/s | 107 kB 00:00 2024-09-21T21:17:36.424 INFO:teuthology.orchestra.run.smithi107.stdout:(13/33): libnbd-1.20.2-2.el9.x86_64.rpm 341 kB/s | 171 kB 00:00 2024-09-21T21:17:36.858 INFO:teuthology.orchestra.run.smithi107.stdout:(14/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 370 kB/s | 160 kB 00:00 2024-09-21T21:17:37.200 INFO:teuthology.orchestra.run.smithi107.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-09-21T21:17:37.325 INFO:teuthology.orchestra.run.smithi107.stdout:(16/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9 2.4 MB/s | 10 MB 00:04 2024-09-21T21:17:37.825 INFO:teuthology.orchestra.run.smithi107.stdout:(17/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 585 kB/s | 292 kB 00:00 2024-09-21T21:17:37.859 INFO:teuthology.orchestra.run.smithi107.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2024-09-21T21:17:38.159 INFO:teuthology.orchestra.run.smithi107.stdout:(19/33): python3-prettytable-0.7.2-27.el9.noarc 127 kB/s | 42 kB 00:00 2024-09-21T21:17:38.343 INFO:teuthology.orchestra.run.smithi107.stdout:(20/33): qatlib-24.02.0-1.el9.x86_64.rpm 456 kB/s | 221 kB 00:00 2024-09-21T21:17:38.418 INFO:teuthology.orchestra.run.smithi107.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 139 kB/s | 36 kB 00:00 2024-09-21T21:17:38.493 INFO:teuthology.orchestra.run.smithi107.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-21T21:17:38.627 INFO:teuthology.orchestra.run.smithi107.stdout:(23/33): libarrow-9.0.0-11.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-09-21T21:17:38.652 INFO:teuthology.orchestra.run.smithi107.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-21T21:17:38.677 INFO:teuthology.orchestra.run.smithi107.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T21:17:38.702 INFO:teuthology.orchestra.run.smithi107.stdout:(26/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 132 kB/s | 47 kB 00:00 2024-09-21T21:17:38.886 INFO:teuthology.orchestra.run.smithi107.stdout:(27/33): ceph-common-19.3.0-5092.g62da54b4.el9. 3.3 MB/s | 20 MB 00:05 2024-09-21T21:17:38.919 INFO:teuthology.orchestra.run.smithi107.stdout:(28/33): libunwind-1.6.2-1.el9.x86_64.rpm 278 kB/s | 67 kB 00:00 2024-09-21T21:17:38.987 INFO:teuthology.orchestra.run.smithi107.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 2.0 MB/s | 191 kB 00:00 2024-09-21T21:17:39.020 INFO:teuthology.orchestra.run.smithi107.stdout:(30/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.6 MB/s | 839 kB 00:00 2024-09-21T21:17:39.079 INFO:teuthology.orchestra.run.smithi107.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-09-21T21:17:39.537 INFO:teuthology.orchestra.run.smithi107.stdout:(32/33): librados2-19.3.0-5092.g62da54b4.el9.x8 6.3 MB/s | 3.5 MB 00:00 2024-09-21T21:17:40.187 INFO:teuthology.orchestra.run.smithi107.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 3.3 MB/s | 3.8 MB 00:01 2024-09-21T21:17:40.193 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:17:40.193 INFO:teuthology.orchestra.run.smithi107.stdout:Total 8.0 MB/s | 60 MB 00:07 2024-09-21T21:17:40.311 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:17:40.348 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:17:40.348 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:17:41.047 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:17:41.048 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:17:42.585 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:17:43.101 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-21T21:17:43.164 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:17:43.256 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:17:43.354 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:17:43.382 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:17:43.445 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:17:43.483 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-21T21:17:43.512 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-21T21:17:43.537 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-21T21:17:43.559 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:17:43.586 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-21T21:17:43.770 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:17:43.801 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-21T21:17:43.836 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-21T21:17:43.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-21T21:17:43.918 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:17:44.016 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-21T21:17:44.039 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:17:44.129 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-21T21:17:44.172 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-21T21:17:44.284 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-21T21:17:44.314 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-21T21:17:44.362 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-21T21:17:44.404 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-21T21:17:44.442 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-21T21:17:44.479 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-21T21:17:44.849 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-21T21:17:44.904 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:17:44.995 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-21T21:17:45.025 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-21T21:17:45.073 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-21T21:17:45.656 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-21T21:17:45.692 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:17:45.763 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-21T21:17:45.845 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-21T21:17:45.878 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-21T21:17:48.333 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:17:48.374 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:17:49.071 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-21T21:17:49.111 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:17:49.194 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-21T21:17:49.194 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T21:17:49.194 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T21:17:49.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:17:49.229 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:17:59.300 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:17:59.300 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-09-21T21:17:59.300 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-09-21T21:17:59.301 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:00.427 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:18:00.452 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-21T21:18:00.452 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:18:00.452 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T21:18:00.452 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:18:00.453 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T21:18:00.453 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:00.454 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:18:00.537 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-21T21:18:00.538 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:18:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:18:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-21T21:18:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-21T21:18:01.892 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-21T21:18:01.893 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-21T21:18:01.894 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-21T21:18:01.895 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-21T21:18:02.907 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:18:02.908 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T21:18:02.909 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.909 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.909 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.909 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.909 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T21:18:02.910 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T21:18:02.911 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T21:18:02.912 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T21:18:02.913 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T21:18:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T21:18:02.914 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:02.914 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:18:03.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-09-21T21:18:03.860 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:33 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:18:03.990 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:18:03.991 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:03.991 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:18:03.991 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:03.991 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:18:03.991 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout:Install 9 Packages 2024-09-21T21:18:03.992 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:03.993 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 83 M 2024-09-21T21:18:03.993 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 775 M 2024-09-21T21:18:03.993 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:18:04.210 INFO:teuthology.orchestra.run.smithi107.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 806 kB/s | 94 kB 00:00 2024-09-21T21:18:04.402 INFO:teuthology.orchestra.run.smithi107.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 566 kB/s | 175 kB 00:00 2024-09-21T21:18:05.053 INFO:teuthology.orchestra.run.smithi107.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 610 kB/s | 513 kB 00:00 2024-09-21T21:18:05.120 INFO:teuthology.orchestra.run.smithi107.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 255 kB/s | 183 kB 00:00 2024-09-21T21:18:05.895 INFO:teuthology.orchestra.run.smithi107.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 392 kB/s | 304 kB 00:00 2024-09-21T21:18:06.045 INFO:teuthology.orchestra.run.smithi107.stdout:(6/9): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-09-21T21:18:06.104 INFO:teuthology.orchestra.run.smithi107.stdout:(7/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.1 MB/s | 122 kB 00:00 2024-09-21T21:18:06.521 INFO:teuthology.orchestra.run.smithi107.stdout:(8/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 102 kB/s | 64 kB 00:00 2024-09-21T21:18:07.621 INFO:teuthology.orchestra.run.smithi107.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 23 MB/s | 81 MB 00:03 2024-09-21T21:18:07.624 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:18:07.624 INFO:teuthology.orchestra.run.smithi107.stdout:Total 23 MB/s | 83 MB 00:03 2024-09-21T21:18:07.709 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:18:07.731 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:18:07.731 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:18:08.325 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:18:08.326 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:18:09.185 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:18:09.267 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-21T21:18:09.333 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-21T21:18:09.386 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-21T21:18:09.420 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-21T21:18:09.467 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:18:09.491 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:18:09.561 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-21T21:18:09.608 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-21T21:18:09.643 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:18:20.426 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-21T21:18:20.517 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:18:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-21T21:18:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-21T21:18:26.554 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-21T21:18:26.555 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-21T21:18:26.974 INFO:teuthology.orchestra.run.smithi107.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:26.975 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:18:27.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-09-21T21:18:27.924 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:18:28.059 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-21T21:18:28.062 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T21:18:28.063 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T21:18:28.064 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout:Install 37 Packages 2024-09-21T21:18:28.065 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:28.066 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 84 M 2024-09-21T21:18:28.066 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 896 M 2024-09-21T21:18:28.066 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:18:28.905 INFO:teuthology.orchestra.run.smithi107.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 44 kB/s | 6.5 kB 00:00 2024-09-21T21:18:29.089 INFO:teuthology.orchestra.run.smithi107.stdout:(2/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 4.8 MB/s | 1.6 MB 00:00 2024-09-21T21:18:29.139 INFO:teuthology.orchestra.run.smithi107.stdout:(3/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 5.7 MB/s | 2.2 MB 00:00 2024-09-21T21:18:29.223 INFO:teuthology.orchestra.run.smithi107.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 3.4 MB/s | 292 kB 00:00 2024-09-21T21:18:29.340 INFO:teuthology.orchestra.run.smithi107.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-21T21:18:29.423 INFO:teuthology.orchestra.run.smithi107.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-21T21:18:29.490 INFO:teuthology.orchestra.run.smithi107.stdout:(7/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 8.7 MB/s | 5.1 MB 00:00 2024-09-21T21:18:29.515 INFO:teuthology.orchestra.run.smithi107.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-21T21:18:29.558 INFO:teuthology.orchestra.run.smithi107.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-21T21:18:29.599 INFO:teuthology.orchestra.run.smithi107.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 5.1 MB/s | 215 kB 00:00 2024-09-21T21:18:29.633 INFO:teuthology.orchestra.run.smithi107.stdout:(11/37): python3-pycparser-2.20-6.el9.noarch.rp 949 kB/s | 135 kB 00:00 2024-09-21T21:18:30.383 INFO:teuthology.orchestra.run.smithi107.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 240 kB/s | 188 kB 00:00 2024-09-21T21:18:30.408 INFO:teuthology.orchestra.run.smithi107.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 222 kB/s | 172 kB 00:00 2024-09-21T21:18:31.067 INFO:teuthology.orchestra.run.smithi107.stdout:(14/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 36 MB/s | 70 MB 00:01 2024-09-21T21:18:31.484 INFO:teuthology.orchestra.run.smithi107.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 32 kB/s | 35 kB 00:01 2024-09-21T21:18:31.626 INFO:teuthology.orchestra.run.smithi107.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 63 kB/s | 77 kB 00:01 2024-09-21T21:18:31.677 INFO:teuthology.orchestra.run.smithi107.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 84 kB/s | 51 kB 00:00 2024-09-21T21:18:31.744 INFO:teuthology.orchestra.run.smithi107.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-21T21:18:31.769 INFO:teuthology.orchestra.run.smithi107.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-21T21:18:31.802 INFO:teuthology.orchestra.run.smithi107.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.0 MB/s | 172 kB 00:00 2024-09-21T21:18:31.928 INFO:teuthology.orchestra.run.smithi107.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-09-21T21:18:31.953 INFO:teuthology.orchestra.run.smithi107.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 68 kB/s | 22 kB 00:00 2024-09-21T21:18:31.979 INFO:teuthology.orchestra.run.smithi107.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-09-21T21:18:32.012 INFO:teuthology.orchestra.run.smithi107.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 893 kB/s | 23 kB 00:00 2024-09-21T21:18:32.037 INFO:teuthology.orchestra.run.smithi107.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 845 kB/s | 19 kB 00:00 2024-09-21T21:18:32.054 INFO:teuthology.orchestra.run.smithi107.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 175 kB/s | 18 kB 00:00 2024-09-21T21:18:32.071 INFO:teuthology.orchestra.run.smithi107.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-21T21:18:32.088 INFO:teuthology.orchestra.run.smithi107.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-09-21T21:18:32.113 INFO:teuthology.orchestra.run.smithi107.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-09-21T21:18:32.130 INFO:teuthology.orchestra.run.smithi107.stdout:(30/37): python3-toml-0.10.2-6.el9.noarch.rpm 64 kB/s | 42 kB 00:00 2024-09-21T21:18:32.147 INFO:teuthology.orchestra.run.smithi107.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 508 kB/s | 16 kB 00:00 2024-09-21T21:18:32.172 INFO:teuthology.orchestra.run.smithi107.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.0 MB/s | 36 kB 00:00 2024-09-21T21:18:32.198 INFO:teuthology.orchestra.run.smithi107.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-21T21:18:32.223 INFO:teuthology.orchestra.run.smithi107.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.6 MB/s | 230 kB 00:00 2024-09-21T21:18:32.282 INFO:teuthology.orchestra.run.smithi107.stdout:(35/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 595 kB/s | 90 kB 00:00 2024-09-21T21:18:32.307 INFO:teuthology.orchestra.run.smithi107.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 240 kB/s | 20 kB 00:00 2024-09-21T21:18:32.340 INFO:teuthology.orchestra.run.smithi107.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-09-21T21:18:32.345 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:18:32.345 INFO:teuthology.orchestra.run.smithi107.stdout:Total 20 MB/s | 84 MB 00:04 2024-09-21T21:18:32.495 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:18:32.537 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:18:32.537 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:18:33.329 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:18:33.330 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:18:34.975 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:18:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T21:18:35.076 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T21:18:35.104 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T21:18:35.176 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T21:18:35.211 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T21:18:35.249 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T21:18:35.340 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T21:18:35.418 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T21:18:35.459 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T21:18:35.495 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T21:18:35.533 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T21:18:35.568 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T21:18:35.601 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T21:18:35.650 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T21:18:35.693 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T21:18:35.727 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T21:18:35.775 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T21:18:35.808 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T21:18:35.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T21:18:36.018 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T21:18:36.100 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T21:18:36.153 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T21:18:36.200 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T21:18:36.257 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T21:18:36.312 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T21:18:36.497 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T21:18:36.546 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T21:18:36.623 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T21:18:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T21:18:36.770 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T21:18:36.896 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T21:18:37.090 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-21T21:18:37.123 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T21:18:37.149 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:47.647 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:18:47.674 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-21T21:18:47.674 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:18:47.674 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T21:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T21:18:47.675 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:48.214 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T21:18:48.238 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:48.482 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:18:48.505 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-21T21:18:48.505 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:18:48.505 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T21:18:48.506 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:18:48.506 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T21:18:48.506 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:48.645 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:18:55.322 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-21T21:18:55.322 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-21T21:18:55.322 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-21T21:18:55.322 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-21T21:18:55.322 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T21:18:55.323 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T21:18:55.324 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T21:18:55.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T21:18:55.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T21:18:55.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T21:18:55.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T21:18:55.326 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T21:18:55.327 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T21:18:56.236 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T21:18:56.237 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T21:18:56.238 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T21:18:56.239 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T21:18:56.239 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:56.239 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:18:56.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-09-21T21:18:57.155 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:18:57.247 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:18:57.290 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:18:57.291 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:18:57.292 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:18:57.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-09-21T21:18:57.973 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:18:58.103 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:18:58.104 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:58.104 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:18:58.104 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 784 k 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 791 k 2024-09-21T21:18:58.105 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:18:58.431 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.4 MB/s | 784 kB 00:00 2024-09-21T21:18:58.431 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:18:58.432 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.3 MB/s | 784 kB 00:00 2024-09-21T21:18:58.432 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:18:58.438 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:18:58.438 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:18:58.443 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:18:58.444 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:18:58.517 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:18:58.995 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:18:59.093 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:18:59.463 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:18:59.712 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:18:59.713 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:59.713 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:18:59.713 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:18:59.713 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:18:59.713 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:18:59.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T21:19:00.547 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:00.684 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:00.684 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:00.684 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 158 k 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 644 k 2024-09-21T21:19:00.685 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:00.896 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 750 kB/s | 158 kB 00:00 2024-09-21T21:19:00.897 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:00.897 INFO:teuthology.orchestra.run.smithi107.stdout:Total 745 kB/s | 158 kB 00:00 2024-09-21T21:19:00.897 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:00.906 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:00.963 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:00.964 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:01.112 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:01.172 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:19:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:19:01.196 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:19:01.196 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T21:19:01.196 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:02.138 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-21T21:19:02.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:02.139 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:02.139 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:02.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:02.139 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:02.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-09-21T21:19:02.990 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:03.081 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:03.123 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:03.124 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:03.124 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:03.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T21:19:03.783 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:03.915 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-21T21:19:03.916 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout:Install 11 Packages 2024-09-21T21:19:03.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:03.918 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 18 M 2024-09-21T21:19:03.918 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 156 M 2024-09-21T21:19:03.918 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:04.360 INFO:teuthology.orchestra.run.smithi107.stdout:(1/11): ceph-prometheus-alerts-19.3.0-5092.g62d 111 kB/s | 16 kB 00:00 2024-09-21T21:19:04.385 INFO:teuthology.orchestra.run.smithi107.stdout:(2/11): ceph-grafana-dashboards-19.3.0-5092.g62 197 kB/s | 34 kB 00:00 2024-09-21T21:19:04.519 INFO:teuthology.orchestra.run.smithi107.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-21T21:19:04.586 INFO:teuthology.orchestra.run.smithi107.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-21T21:19:04.620 INFO:teuthology.orchestra.run.smithi107.stdout:(5/11): protobuf-compiler-3.14.0-13.el9.x86_64. 3.6 MB/s | 863 kB 00:00 2024-09-21T21:19:04.636 INFO:teuthology.orchestra.run.smithi107.stdout:(6/11): grpc-data-1.46.7-10.el9.noarch.rpm 388 kB/s | 19 kB 00:00 2024-09-21T21:19:04.670 INFO:teuthology.orchestra.run.smithi107.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.4 MB/s | 144 kB 00:00 2024-09-21T21:19:04.687 INFO:teuthology.orchestra.run.smithi107.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 2.0 MB/s | 31 kB 00:00 2024-09-21T21:19:04.750 INFO:teuthology.orchestra.run.smithi107.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 3.0 MB/s | 188 kB 00:00 2024-09-21T21:19:04.938 INFO:teuthology.orchestra.run.smithi107.stdout:(10/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 20 MB/s | 14 MB 00:00 2024-09-21T21:19:04.996 INFO:teuthology.orchestra.run.smithi107.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.4 MB/s | 2.0 MB 00:00 2024-09-21T21:19:05.000 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:05.000 INFO:teuthology.orchestra.run.smithi107.stdout:Total 17 MB/s | 18 MB 00:01 2024-09-21T21:19:05.029 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:05.048 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:05.049 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:05.304 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:05.304 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:05.863 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:05.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-21T21:19:05.983 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-21T21:19:06.089 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-21T21:19:06.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-21T21:19:06.261 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-21T21:19:06.345 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-21T21:19:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-21T21:19:06.417 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:19:06.450 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-21T21:19:09.641 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-21T21:19:09.713 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:19:11.588 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-21T21:19:11.589 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:11.904 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:11.905 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:12.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T21:19:12.780 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:12.915 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T21:19:12.916 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-09-21T21:19:12.917 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:12.918 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 38 M 2024-09-21T21:19:12.918 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 212 M 2024-09-21T21:19:12.918 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:13.304 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-21T21:19:13.404 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 328 kB/s | 33 kB 00:00 2024-09-21T21:19:13.479 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.2 MB/s | 801 kB 00:00 2024-09-21T21:19:13.563 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 219 kB/s | 18 kB 00:00 2024-09-21T21:19:13.630 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 577 kB/s | 38 kB 00:00 2024-09-21T21:19:13.705 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-09-21T21:19:13.755 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-21T21:19:13.880 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-09-21T21:19:13.956 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-21T21:19:13.981 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-09-21T21:19:14.256 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 54 MB/s | 16 MB 00:00 2024-09-21T21:19:14.890 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): ceph-mgr-diskprediction-local-19.3.0-5 4.2 MB/s | 7.4 MB 00:01 2024-09-21T21:19:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:14.891 INFO:teuthology.orchestra.run.smithi107.stdout:Total 19 MB/s | 38 MB 00:01 2024-09-21T21:19:15.247 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:15.273 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:15.274 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:15.696 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:15.696 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:16.447 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:16.523 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T21:19:16.563 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:19:16.662 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:19:16.698 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T21:19:16.882 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T21:19:16.913 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T21:19:16.997 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T21:19:17.650 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T21:19:17.768 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T21:19:19.012 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:19:20.040 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T21:19:20.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T21:19:22.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T21:19:22.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T21:19:22.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T21:19:22.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T21:19:22.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T21:19:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T21:19:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:23.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:23.193 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T21:19:23.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:23.195 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:23.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-09-21T21:19:24.237 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:54 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:24.370 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T21:19:24.371 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:24.372 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-09-21T21:19:24.373 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:24.373 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.2 M 2024-09-21T21:19:24.373 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-09-21T21:19:24.373 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:24.815 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 292 kB/s | 49 kB 00:00 2024-09-21T21:19:24.915 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 71 kB/s | 19 kB 00:00 2024-09-21T21:19:24.940 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 87 kB/s | 26 kB 00:00 2024-09-21T21:19:25.341 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 423 kB/s | 222 kB 00:00 2024-09-21T21:19:25.400 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 328 kB/s | 159 kB 00:00 2024-09-21T21:19:25.441 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 775 kB/s | 32 kB 00:00 2024-09-21T21:19:25.467 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 530 kB/s | 279 kB 00:00 2024-09-21T21:19:25.492 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 280 kB/s | 14 kB 00:00 2024-09-21T21:19:25.551 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 2.6 MB/s | 220 kB 00:00 2024-09-21T21:19:25.576 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T21:19:25.618 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.6 MB/s | 1.0 MB 00:00 2024-09-21T21:19:25.643 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-21T21:19:25.677 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 160 kB/s | 54 kB 00:00 2024-09-21T21:19:25.680 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:25.680 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-21T21:19:25.750 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:25.765 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:25.853 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:25.854 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:26.381 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:26.556 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T21:19:26.619 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T21:19:26.684 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T21:19:26.721 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T21:19:26.762 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T21:19:26.864 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T21:19:26.975 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T21:19:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T21:19:27.748 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T21:19:27.803 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T21:19:27.837 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T21:19:27.886 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T21:19:27.907 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T21:19:28.702 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T21:19:28.703 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T21:19:29.037 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:29.038 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:29.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T21:19:29.916 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:59 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:30.048 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:30.048 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:30.049 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-09-21T21:19:30.050 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:30.050 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 7.0 M 2024-09-21T21:19:30.050 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-09-21T21:19:30.050 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:30.381 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 689 kB/s | 154 kB 00:00 2024-09-21T21:19:30.448 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-09-21T21:19:30.473 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T21:19:30.507 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-21T21:19:30.715 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 445 kB/s | 248 kB 00:00 2024-09-21T21:19:31.382 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.9 MB/s | 6.0 MB 00:01 2024-09-21T21:19:31.384 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:31.384 INFO:teuthology.orchestra.run.smithi107.stdout:Total 5.3 MB/s | 7.0 MB 00:01 2024-09-21T21:19:31.453 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:31.465 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:31.465 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:31.565 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:31.571 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:31.879 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:32.048 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T21:19:32.100 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T21:19:32.616 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T21:19:32.702 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T21:19:32.779 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T21:19:32.813 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T21:19:33.688 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T21:19:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T21:19:33.968 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T21:19:33.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:33.968 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:34.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-09-21T21:19:34.851 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:04 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:34.983 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:34.983 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:34.983 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:34.983 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-09-21T21:19:34.984 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:34.985 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 963 k 2024-09-21T21:19:34.985 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 4.1 M 2024-09-21T21:19:34.985 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:35.310 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.1 MB/s | 883 kB 00:00 2024-09-21T21:19:35.494 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 173 kB/s | 80 kB 00:00 2024-09-21T21:19:35.494 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:35.494 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.8 MB/s | 963 kB 00:00 2024-09-21T21:19:35.506 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:35.530 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:35.530 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:35.610 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:35.610 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:35.797 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:35.946 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T21:19:36.017 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:19:36.743 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-21T21:19:36.744 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-21T21:19:37.021 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T21:19:37.021 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:37.021 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:37.022 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T21:19:37.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:37.022 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:37.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2024-09-21T21:19:37.903 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:07 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:38.039 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:38.040 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:38.041 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 285 k 2024-09-21T21:19:38.041 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.3 M 2024-09-21T21:19:38.041 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:38.266 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-21T21:19:38.266 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:38.267 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T21:19:38.267 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:38.275 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:38.275 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:38.311 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:38.311 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:38.530 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:38.601 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:19:38.617 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:19:38.618 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:19:38.618 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T21:19:38.618 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:39.654 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-21T21:19:39.655 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:39.655 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:39.655 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-21T21:19:39.655 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:39.655 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:39.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-09-21T21:19:40.567 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:10 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:40.703 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:40.758 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:40.758 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T21:19:40.758 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:40.758 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 125 k 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 445 k 2024-09-21T21:19:40.759 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:40.913 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 599 kB/s | 125 kB 00:00 2024-09-21T21:19:40.914 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:40.915 INFO:teuthology.orchestra.run.smithi107.stdout:Total 595 kB/s | 125 kB 00:00 2024-09-21T21:19:40.915 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:40.920 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:40.920 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:40.985 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:40.985 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:41.146 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:41.252 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:41.710 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:41.978 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:42.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-09-21T21:19:42.862 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:12 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:42.954 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:42.999 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:42.999 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:43.000 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:43.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-09-21T21:19:43.684 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:13 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:43.818 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:43.818 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:43.818 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T21:19:43.818 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:43.818 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 32 k 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 142 k 2024-09-21T21:19:43.819 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 213 kB/s | 32 kB 00:00 2024-09-21T21:19:43.967 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-21T21:19:43.968 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:43.971 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:43.986 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:43.987 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:44.075 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:44.197 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:44.579 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:44.804 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:44.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:44.805 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:44.805 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:44.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:44.805 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:45.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-09-21T21:19:45.587 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:15 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:45.676 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:45.719 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:45.720 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:45.720 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:45.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-09-21T21:19:46.385 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:16 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:46.472 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:46.515 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:46.516 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:46.516 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:46.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-09-21T21:19:47.188 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:17 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:47.277 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:47.320 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:47.321 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:47.321 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:47.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-09-21T21:19:47.985 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:17 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:48.073 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:48.117 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:48.118 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:48.118 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:48.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-09-21T21:19:48.780 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:18 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:48.868 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:48.912 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:48.913 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:48.913 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:48.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-09-21T21:19:49.569 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:19 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:49.656 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-21T21:19:49.700 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:49.701 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-21T21:19:49.701 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:49.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-09-21T21:19:50.384 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:20 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:50.517 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:50.518 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:50.518 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T21:19:50.518 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:50.518 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 91 k 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 321 k 2024-09-21T21:19:50.519 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:50.720 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 452 kB/s | 91 kB 00:00 2024-09-21T21:19:50.721 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:50.721 INFO:teuthology.orchestra.run.smithi107.stdout:Total 449 kB/s | 91 kB 00:00 2024-09-21T21:19:50.721 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:50.730 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:50.730 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:50.794 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:50.794 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:50.946 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:51.044 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:51.469 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:51.713 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:51.714 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:51.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:51.714 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:51.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-09-21T21:19:52.587 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:22 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:52.722 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:52.723 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:52.724 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:52.724 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.5 M 2024-09-21T21:19:52.724 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 30 M 2024-09-21T21:19:52.724 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:53.041 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2024-09-21T21:19:53.041 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:53.042 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 3.5 MB 00:00 2024-09-21T21:19:53.042 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:53.053 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:53.053 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:53.136 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:53.136 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:53.713 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:53.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:53.809 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:53.809 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T21:19:53.809 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T21:19:53.809 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:19:53.809 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T21:19:53.810 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:55.100 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:55.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-09-21T21:19:55.963 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:25 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:19:56.095 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-21T21:19:56.096 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:56.097 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 176 k 2024-09-21T21:19:56.097 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 685 k 2024-09-21T21:19:56.097 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:19:56.287 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 932 kB/s | 176 kB 00:00 2024-09-21T21:19:56.287 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:19:56.287 INFO:teuthology.orchestra.run.smithi107.stdout:Total 924 kB/s | 176 kB 00:00 2024-09-21T21:19:56.288 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:19:56.297 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:19:56.297 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:19:56.360 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:19:56.360 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:19:56.515 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:19:56.610 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:57.046 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:57.329 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:19:57.651 DEBUG:teuthology.parallel:result is None 2024-09-21T21:19:57.652 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T21:19:57.800 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T21:19:57.822 INFO:teuthology.orchestra.run.smithi107.stdout:19.3.0-5092.g62da54b4.el9 2024-09-21T21:19:57.822 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-21T21:19:57.822 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-21T21:19:57.824 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T21:19:57.824 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:57.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T21:19:57.888 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T21:19:57.888 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:57.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T21:19:57.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T21:19:58.018 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T21:19:58.018 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:58.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T21:19:58.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T21:19:58.147 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T21:19:58.147 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:58.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T21:19:58.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T21:19:58.277 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T21:19:58.351 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T21:19:58.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2024-09-21T21:19:58.376 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T21:19:58.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T21:19:58.440 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T21:19:58.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T21:19:58.511 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T21:19:58.511 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T21:19:58.511 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914337', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914337', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0FTx0oLkzEd2LoLzZYPS3ETIjYjRH/HIlJ7/+EWbU+QuO8JYLOd4vTGAzNJgxCrfW/DVKYcXXo7mKefWuybBA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T21:19:58.512 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T21:19:58.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T21:19:58.624 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:58.624 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T21:19:58.679 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T21:19:58.680 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 21:19:54.072012626 +0000 2024-09-21T21:19:58.735 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 21:16:53.472818729 +0000 2024-09-21T21:19:58.736 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 21:16:53.472818729 +0000 2024-09-21T21:19:58.736 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 21:16:53.472818729 +0000 2024-09-21T21:19:58.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T21:19:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T21:19:58.798 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T21:19:58.799 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000199615 s, 2.6 MB/s 2024-09-21T21:19:58.800 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T21:19:58.857 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-09-21T21:19:58.912 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 21:19:54.073012610 +0000 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 21:16:53.794813547 +0000 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 21:16:53.794813547 +0000 2024-09-21T21:19:58.913 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 21:16:53.794813547 +0000 2024-09-21T21:19:58.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T21:19:58.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T21:19:58.975 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T21:19:58.976 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000243698 s, 2.1 MB/s 2024-09-21T21:19:58.977 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T21:19:59.034 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-09-21T21:19:59.089 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T21:19:59.089 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:19:59.089 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T21:19:59.089 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:19:59.089 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:19:59.090 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 21:19:54.073012610 +0000 2024-09-21T21:19:59.090 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 21:16:54.109808475 +0000 2024-09-21T21:19:59.090 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 21:16:54.109808475 +0000 2024-09-21T21:19:59.090 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 21:16:54.109808475 +0000 2024-09-21T21:19:59.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T21:19:59.152 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T21:19:59.152 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T21:19:59.152 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000206384 s, 2.5 MB/s 2024-09-21T21:19:59.153 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T21:19:59.210 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-09-21 21:19:54.073012610 +0000 2024-09-21T21:19:59.265 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-09-21 21:16:54.427803356 +0000 2024-09-21T21:19:59.266 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-09-21 21:16:54.427803356 +0000 2024-09-21T21:19:59.266 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-09-21 21:16:54.427803356 +0000 2024-09-21T21:19:59.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T21:19:59.328 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-09-21T21:19:59.328 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-09-21T21:19:59.328 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000171674 s, 3.0 MB/s 2024-09-21T21:19:59.329 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T21:19:59.385 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:19:59.385 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:19:59.385 INFO:tasks.ceph:Generating config... 2024-09-21T21:19:59.386 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-21T21:19:59.386 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T21:19:59.386 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-21T21:19:59.386 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] err to stderr = True 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] log flush on exit = True 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] log to file = False 2024-09-21T21:19:59.387 INFO:tasks.ceph:[osd] log to stderr = True 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T21:19:59.388 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] auth client required = none 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] auth service required = none 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] auth supported = none 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-21T21:19:59.388 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-21T21:19:59.389 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-21T21:19:59.390 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-21T21:19:59.391 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-21T21:19:59.392 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-21T21:19:59.393 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-21T21:19:59.393 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-21T21:19:59.393 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-21T21:19:59.394 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-21T21:19:59.394 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-21T21:19:59.394 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T21:19:59.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T21:19:59.477 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T21:19:59.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T21:19:59.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:19:59.555 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.107')] 2024-09-21T21:19:59.556 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.107', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-21T21:19:59.557 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:59.557 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:19:59.610 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.107 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:19:59.701 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:19:59.700+0000 7f41af8dff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:19:59.700+0000 7f41af8dff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:19:59.700+0000 7f41af8dff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid 5cb54416-547e-4ffc-bb2c-85185b12fcd0 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-09-21T21:19:59.702 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 5cb54416-547e-4ffc-bb2c-85185b12fcd0 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2024-09-21T21:19:59.701793+0000 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:created 2024-09-21T21:19:59.701793+0000 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 17 (quincy) 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 1 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:0: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.a 2024-09-21T21:19:59.703 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-21T21:19:59.704 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T21:19:59.719 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5cb54416-547e-4ffc-bb2c-85185b12fcd0... 2024-09-21T21:19:59.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T21:19:59.802 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-09-21T21:19:59.802 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2024-09-21T21:19:59.802 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = none 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T21:19:59.803 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2024-09-21T21:19:59.804 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2024-09-21T21:19:59.805 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = 172.21.15.107 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: auth client required = none 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: auth cluster required = none 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: auth service required = none 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug asok = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug bdev = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug buffer = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug civetweb = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug client = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug compressor = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug context = 0/0 2024-09-21T21:19:59.806 INFO:teuthology.orchestra.run.smithi107.stdout: debug crush = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug crypto = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug dpdk = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug eventtrace = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug filer = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug filestore = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug finisher = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug fuse = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug heartbeatmap = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug javaclient = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug journal = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug journaler = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug kstore = 0/0 2024-09-21T21:19:59.807 INFO:teuthology.orchestra.run.smithi107.stdout: debug leveldb = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug lockdep = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds balancer = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds locker = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds log = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds log expire = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mds migrator = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug memdb = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgrc = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 0/0 2024-09-21T21:19:59.808 INFO:teuthology.orchestra.run.smithi107.stdout: debug objclass = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug objectcacher = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug objecter = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug optracker = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug perfcounter = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug rados = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug rbd = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug rbd mirror = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug rbd replay = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug refs = 0/0 2024-09-21T21:19:59.809 INFO:teuthology.orchestra.run.smithi107.stdout: debug rgw = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug rgw lifecycle = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug rgw notification = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug rgw sync = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug striper = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug throttle = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug timer = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: debug tp = 0/0 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: ms client mode = crc 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: ms cluster mode = crc 2024-09-21T21:19:59.810 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon client mode = crc 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon cluster mode = crc 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: ms mon service mode = crc 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: ms service mode = crc 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 5cb54416-547e-4ffc-bb2c-85185b12fcd0 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.811 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.812 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 20 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression algorithm = snappy 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression mode = aggressive 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2024-09-21T21:19:59.813 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore rocksdb cf = False 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: crimson alien thread cpu cores = 6-7 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: crimson osd obc lru size = 10 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug filestore = 0/0 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug journal = 0/0 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 0/0 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 0/0 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: err to stderr = True 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: log flush on exit = True 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: log to file = False 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: log to stderr = True 2024-09-21T21:19:59.814 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: osd shutdown pgref assert = True 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2024-09-21T21:19:59.815 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 0/0 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 0/0 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 0/0 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T21:19:59.816 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default crimson = True 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:19:59.817 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-09-21T21:19:59.818 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 0-2 2024-09-21T21:19:59.819 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.3] 2024-09-21T21:19:59.819 INFO:teuthology.orchestra.run.smithi107.stdout: crimson seastar cpu cores = 3-5 2024-09-21T21:19:59.819 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T21:19:59.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T21:19:59.925 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T21:19:59.925 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:59.926 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T21:19:59.941 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:59.941 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T21:19:59.996 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T21:19:59.996 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:19:59.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T21:19:59.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T21:20:00.079 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:00.079 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:20:00.134 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T21:20:00.135 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T21:20:00.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:20:00.233 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T21:20:00.237 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T21:20:00.237 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T21:20:00.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T21:20:00.286 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T21:20:00.299 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T21:20:00.299 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-21T21:20:00.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T21:20:00.323 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:20:00.324 INFO:tasks.ceph:role: osd.0 2024-09-21T21:20:00.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T21:20:00.324 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T21:20:00.397 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:20:00.397 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:20:00.397 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:20:00.397 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:20:00.398 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:20:00.908 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-09-21T21:20:00.910 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-09-21T21:20:00.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T21:20:01.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T21:20:01.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T21:20:01.132 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:20:01.133 INFO:tasks.ceph:role: osd.1 2024-09-21T21:20:01.133 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T21:20:01.133 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:20:01.205 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:20:01.677 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-09-21T21:20:01.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-09-21T21:20:01.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T21:20:01.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T21:20:01.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T21:20:01.869 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T21:20:01.869 INFO:tasks.ceph:role: osd.2 2024-09-21T21:20:01.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2024-09-21T21:20:01.869 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2024-09-21T21:20:01.942 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T21:20:01.943 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T21:20:01.943 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T21:20:01.943 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T21:20:02.430 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2024-09-21T21:20:02.432 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2024-09-21T21:20:02.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T21:20:02.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T21:20:02.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:20:02.924 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:02.925 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:02,925 seastar - Reactor backend: linux-aio 2024-09-21T21:20:02.937 INFO:teuthology.orchestra.run.smithi107.stderr:==28117==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:03.068 INFO:teuthology.orchestra.run.smithi107.stderr:DEBUG 2024-09-21 21:20:03,067 [shard 0:main] seastar - reactor::drain 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:03.580 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:03.640 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:03.644 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:03,643 seastar - Reactor backend: linux-aio 2024-09-21T21:20:03.653 INFO:teuthology.orchestra.run.smithi107.stderr:==28115==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:08.315 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:08.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T21:20:08.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:20:08.647 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:08.648 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:08,648 seastar - Reactor backend: linux-aio 2024-09-21T21:20:08.653 INFO:teuthology.orchestra.run.smithi107.stderr:==28303==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:08.784 INFO:teuthology.orchestra.run.smithi107.stderr:DEBUG 2024-09-21 21:20:08,783 [shard 0:main] seastar - reactor::drain 2024-09-21T21:20:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:09.282 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:09.282 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:09.283 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:09.283 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:09.283 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:09.345 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:09.348 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:09,348 seastar - Reactor backend: linux-aio 2024-09-21T21:20:09.365 INFO:teuthology.orchestra.run.smithi107.stderr:==28301==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:14.059 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:14.060 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:14.060 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:14.060 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:14.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T21:20:14.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:20:14.397 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:14.398 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:14,398 seastar - Reactor backend: linux-aio 2024-09-21T21:20:14.403 INFO:teuthology.orchestra.run.smithi107.stderr:==28490==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:14.510 INFO:teuthology.orchestra.run.smithi107.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133cca 0xdd332 0x4b14c 0x5fcd0d5 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-21T21:20:14.510 INFO:teuthology.orchestra.run.smithi107.stderr:kernel callstack: 2024-09-21T21:20:14.534 INFO:teuthology.orchestra.run.smithi107.stderr:DEBUG 2024-09-21 21:20:14,534 [shard 0:main] seastar - reactor::drain 2024-09-21T21:20:15.048 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:15.049 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:15.049 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:15.049 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:15.049 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:15.101 INFO:teuthology.orchestra.run.smithi107.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-21T21:20:15.105 INFO:teuthology.orchestra.run.smithi107.stderr:INFO 2024-09-21 21:20:15,104 seastar - Reactor backend: linux-aio 2024-09-21T21:20:15.122 INFO:teuthology.orchestra.run.smithi107.stderr:==28488==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:Suppressions used: 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr: count bytes template 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr: 1 8 MallocExtension::Register 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr:----------------------------------------------------- 2024-09-21T21:20:19.793 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:19.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T21:20:19.840 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T21:20:19.840 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:19.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T21:20:19.905 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:19.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T21:20:19.969 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:19.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T21:20:20.034 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:20.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T21:20:20.099 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:20.099 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T21:20:20.154 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T21:20:20.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCAOO9m9SzuDRAA2gHENWDYzw/VzwNafijdHg== 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCDOO9m0miPLxAAlU5Qk1GOdI/98hDw+k43cw== 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCJOO9mqXduHhAAYm+VXz8pxg6+Lm/pxco7xg== 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCPOO9m4D7XDxAAay4SHisLwLiqArPYkSoVFw== 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-09-21T21:20:20.220 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQCAOO9m7mUMERAA8DssknCZYyOphGDaTmo+LQ== 2024-09-21T21:20:20.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T21:20:20.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:20:20.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:20:20.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T21:20:20.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T21:20:20.568 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T21:20:20.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T21:20:20.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T21:20:20.719 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:20.718+0000 7f506d0e6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.720 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:20.718+0000 7f506d0e6c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T21:20:20.757 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T21:20:20.813 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T21:20:20.813 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T21:20:20.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T21:20:20.855 INFO:tasks.ceph.mon.a:Started 2024-09-21T21:20:20.855 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T21:20:20.856 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T21:20:20.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T21:20:20.858 INFO:tasks.ceph.mgr.x:Started 2024-09-21T21:20:20.859 DEBUG:tasks.ceph:set 0 configs 2024-09-21T21:20:20.859 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T21:20:20.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T21:20:20.937 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:20:20.935+0000 7f3947c66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.938 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:20:20.935+0000 7f3947c66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.938 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:20:20.936+0000 7f3947c66c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.943 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:20:20.941+0000 7fcd95dc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.970 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:20.962+0000 7f6ecfd51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:20.972 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:20.970+0000 7f6ecfd51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:21.038 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-09-21T21:20:21.049 INFO:tasks.ceph:check_enable_crimson: True 2024-09-21T21:20:21.050 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-21T21:20:21.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-21T21:20:21.147 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:20:21.146+0000 7fcd95dc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:21.147 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:20:21.146+0000 7fcd95dc5000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:21.155 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:21.149+0000 7fbfb6991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:21.157 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:21.155+0000 7fbfb6991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:21.233 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:20:21.232+0000 7f393e5f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-21T21:20:21.233 INFO:tasks.ceph.mon.a.smithi107.stderr:Please be aware that this feature is experimental, untested, 2024-09-21T21:20:21.234 INFO:tasks.ceph.mon.a.smithi107.stderr:unsupported, and may result in data corruption, data loss, 2024-09-21T21:20:21.234 INFO:tasks.ceph.mon.a.smithi107.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-21T21:20:21.234 INFO:tasks.ceph.mon.a.smithi107.stderr:feature with important data. 2024-09-21T21:20:21.234 INFO:tasks.ceph.mon.a.smithi107.stderr: 2024-09-21T21:20:22.050 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T21:20:22.050 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:22.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T21:20:22.076 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:22.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T21:20:22.145 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:22.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T21:20:22.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new aba0b78f-46aa-434d-bda5-0adebbad89c3 0 2024-09-21T21:20:22.364 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.357+0000 7f5ce7991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.367 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.365+0000 7f5ce7991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.441 INFO:teuthology.orchestra.run.smithi107.stdout:0 2024-09-21T21:20:22.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new e430b7fa-508d-43d1-8ffc-0761298fd5b0 1 2024-09-21T21:20:22.562 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.554+0000 7f5fa54d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.564 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.562+0000 7f5fa54d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.640 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-09-21T21:20:22.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new 5f35ad36-4f8f-4272-a660-76dee9729968 2 2024-09-21T21:20:22.760 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.753+0000 7f13870ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.762 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:22.761+0000 7f13870ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:22.839 INFO:teuthology.orchestra.run.smithi107.stdout:2 2024-09-21T21:20:22.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-21T21:20:22.879 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T21:20:22.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-21T21:20:22.922 INFO:tasks.ceph.osd.0:Started 2024-09-21T21:20:22.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-21T21:20:22.953 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T21:20:22.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-21T21:20:22.996 INFO:tasks.ceph.osd.1:Started 2024-09-21T21:20:22.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-21T21:20:23.027 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T21:20:23.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-21T21:20:23.070 INFO:tasks.ceph.osd.2:Started 2024-09-21T21:20:23.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:20:23.230 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:23.224+0000 7fcf27f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:23.233 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:23.231+0000 7fcf27f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:23.301 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:23.301 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":6,"fsid":"5cb54416-547e-4ffc-bb2c-85185b12fcd0","created":"2024-09-21T21:20:20.959278+0000","modified":"2024-09-21T21:20:22.838001+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:20:22.838001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"aba0b78f-46aa-434d-bda5-0adebbad89c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e430b7fa-508d-43d1-8ffc-0761298fd5b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5f35ad36-4f8f-4272-a660-76dee9729968","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:20:23.317 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T21:20:23.317 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T21:20:23.618 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:20:23.723 INFO:teuthology.misc.health.smithi107.stderr:2024-09-21T21:20:23.717+0000 7f1a79f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:23.725 INFO:teuthology.misc.health.smithi107.stderr:2024-09-21T21:20:23.724+0000 7f1a79f51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:23.792 INFO:teuthology.misc.health.smithi107.stdout: 2024-09-21T21:20:23.792 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":6,"fsid":"5cb54416-547e-4ffc-bb2c-85185b12fcd0","created":"2024-09-21T21:20:20.959278+0000","modified":"2024-09-21T21:20:22.838001+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T21:20:22.838001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"aba0b78f-46aa-434d-bda5-0adebbad89c3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e430b7fa-508d-43d1-8ffc-0761298fd5b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5f35ad36-4f8f-4272-a660-76dee9729968","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:20:23.803 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-21T21:20:26.495 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:20:26.493+0000 7fcd32f93640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T21:20:30.105 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T21:20:30.213 INFO:teuthology.misc.health.smithi107.stderr:2024-09-21T21:20:30.208+0000 7f15ce191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:30.215 INFO:teuthology.misc.health.smithi107.stderr:2024-09-21T21:20:30.213+0000 7f15ce191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:30.405 INFO:teuthology.misc.health.smithi107.stdout: 2024-09-21T21:20:30.405 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":8,"fsid":"5cb54416-547e-4ffc-bb2c-85185b12fcd0","created":"2024-09-21T21:20:20.959278+0000","modified":"2024-09-21T21:20:29.500488+0000","last_up_change":"2024-09-21T21:20:29.500488+0000","last_in_change":"2024-09-21T21:20:22.838001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"aba0b78f-46aa-434d-bda5-0adebbad89c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":3908228775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3908228775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3908228775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3908228775}]},"public_addr":"172.21.15.107:6802/3908228775","cluster_addr":"172.21.15.107:6805/3908228775","heartbeat_back_addr":"172.21.15.107:6809/3908228775","heartbeat_front_addr":"172.21.15.107:6808/3908228775","state":["exists","up"]},{"osd":1,"uuid":"e430b7fa-508d-43d1-8ffc-0761298fd5b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3406272017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":3406272017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3406272017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":3406272017}]},"public_addr":"172.21.15.107:6804/3406272017","cluster_addr":"172.21.15.107:6803/3406272017","heartbeat_back_addr":"172.21.15.107:6812/3406272017","heartbeat_front_addr":"172.21.15.107:6813/3406272017","state":["exists","up"]},{"osd":2,"uuid":"5f35ad36-4f8f-4272-a660-76dee9729968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":3042772323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":3042772323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3042772323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":3042772323}]},"public_addr":"172.21.15.107:6806/3042772323","cluster_addr":"172.21.15.107:6807/3042772323","heartbeat_back_addr":"172.21.15.107:6816/3042772323","heartbeat_front_addr":"172.21.15.107:6817/3042772323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:20:30.417 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-21T21:20:30.417 INFO:tasks.ceph:Creating RBD pool 2024-09-21T21:20:30.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T21:20:30.526 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:30.517+0000 7f51ed551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:30.528 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:30.527+0000 7f51ed551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:31.513 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2024-09-21T21:20:31.533 DEBUG:teuthology.orchestra.run.smithi107:> rbd --cluster ceph pool init rbd 2024-09-21T21:20:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:31.629+0000 7f690aa5aac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:31.630 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:31.629+0000 7f690aa5aac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:31.632 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:31.631+0000 7f690aa5aac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:34.566 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T21:20:34.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T21:20:34.567 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T21:20:34.677 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:34.671+0000 7f8c84591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:34.679 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:34.678+0000 7f8c84591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:34.870 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:34.882 INFO:teuthology.orchestra.run.smithi107.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-21T21:20:34.882 INFO:tasks.ceph_manager:config epoch is 1 2024-09-21T21:20:34.882 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T21:20:34.883 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T21:20:34.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T21:20:34.991 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:34.986+0000 7f2fc046c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:34.993 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:34.992+0000 7f2fc046c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.230 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:35.243 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1862686799},{"type":"v1","addr":"172.21.15.107:6801","nonce":1862686799}]},"active_addr":"172.21.15.107:6801/1862686799","active_change":"2024-09-21T21:20:25.486496+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":2954240977}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":507316432}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1883777343}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1736570364}]}]} 2024-09-21T21:20:35.245 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T21:20:35.245 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T21:20:35.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:20:35.352 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.347+0000 7f21b38ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.354 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.353+0000 7f21b38ac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.542 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:35.543 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"fsid":"5cb54416-547e-4ffc-bb2c-85185b12fcd0","created":"2024-09-21T21:20:20.959278+0000","modified":"2024-09-21T21:20:34.521715+0000","last_up_change":"2024-09-21T21:20:29.500488+0000","last_in_change":"2024-09-21T21:20:22.838001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:20:30.719687+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:20:31.504769+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aba0b78f-46aa-434d-bda5-0adebbad89c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":3908228775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3908228775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3908228775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3908228775}]},"public_addr":"172.21.15.107:6802/3908228775","cluster_addr":"172.21.15.107:6805/3908228775","heartbeat_back_addr":"172.21.15.107:6809/3908228775","heartbeat_front_addr":"172.21.15.107:6808/3908228775","state":["exists","up"]},{"osd":1,"uuid":"e430b7fa-508d-43d1-8ffc-0761298fd5b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3406272017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":3406272017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3406272017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":3406272017}]},"public_addr":"172.21.15.107:6804/3406272017","cluster_addr":"172.21.15.107:6803/3406272017","heartbeat_back_addr":"172.21.15.107:6812/3406272017","heartbeat_front_addr":"172.21.15.107:6813/3406272017","state":["exists","up"]},{"osd":2,"uuid":"5f35ad36-4f8f-4272-a660-76dee9729968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":3042772323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":3042772323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3042772323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":3042772323}]},"public_addr":"172.21.15.107:6806/3042772323","cluster_addr":"172.21.15.107:6807/3042772323","heartbeat_back_addr":"172.21.15.107:6816/3042772323","heartbeat_front_addr":"172.21.15.107:6817/3042772323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:20:35.555 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T21:20:35.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T21:20:35.664 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.659+0000 7f88eb8fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.666 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.665+0000 7f88eb8fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.854 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:35.854 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":12,"fsid":"5cb54416-547e-4ffc-bb2c-85185b12fcd0","created":"2024-09-21T21:20:20.959278+0000","modified":"2024-09-21T21:20:34.521715+0000","last_up_change":"2024-09-21T21:20:29.500488+0000","last_in_change":"2024-09-21T21:20:22.838001+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-21T21:20:30.719687+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":12,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-21T21:20:31.504769+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aba0b78f-46aa-434d-bda5-0adebbad89c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":3908228775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6805","nonce":3908228775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6809","nonce":3908228775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":3908228775}]},"public_addr":"172.21.15.107:6802/3908228775","cluster_addr":"172.21.15.107:6805/3908228775","heartbeat_back_addr":"172.21.15.107:6809/3908228775","heartbeat_front_addr":"172.21.15.107:6808/3908228775","state":["exists","up"]},{"osd":1,"uuid":"e430b7fa-508d-43d1-8ffc-0761298fd5b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":3406272017}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6803","nonce":3406272017}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":3406272017}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6813","nonce":3406272017}]},"public_addr":"172.21.15.107:6804/3406272017","cluster_addr":"172.21.15.107:6803/3406272017","heartbeat_back_addr":"172.21.15.107:6812/3406272017","heartbeat_front_addr":"172.21.15.107:6813/3406272017","state":["exists","up"]},{"osd":2,"uuid":"5f35ad36-4f8f-4272-a660-76dee9729968","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":3042772323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6807","nonce":3042772323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":3042772323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6817","nonce":3042772323}]},"public_addr":"172.21.15.107:6806/3042772323","cluster_addr":"172.21.15.107:6807/3042772323","heartbeat_back_addr":"172.21.15.107:6816/3042772323","heartbeat_front_addr":"172.21.15.107:6817/3042772323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T21:20:35.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T21:20:35.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T21:20:35.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T21:20:35.981 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.975+0000 7f3155c93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.975+0000 7f1cc4ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.982 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.975+0000 7f58212d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.982+0000 7f58212d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.983+0000 7f3155c93640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:35.984 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:35.983+0000 7f1cc4ea3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.013 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T21:20:36.014 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T21:20:36.018 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T21:20:36.130 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.122+0000 7fd4fdd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.133 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.131+0000 7fd4fdd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.165+0000 7efdc1e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.173 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.167+0000 7f12384fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.174 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.173+0000 7efdc1e23640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.176 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.175+0000 7f12384fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.355 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.2 2024-09-21T21:20:36.368 DEBUG:teuthology.parallel:result is None 2024-09-21T21:20:36.376 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.1 2024-09-21T21:20:36.389 DEBUG:teuthology.parallel:result is None 2024-09-21T21:20:36.389 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2024-09-21T21:20:36.403 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738374 got 34359738374 for osd.0 2024-09-21T21:20:36.403 DEBUG:teuthology.parallel:result is None 2024-09-21T21:20:36.403 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T21:20:36.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:20:36.552 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.547+0000 7f9628351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.554 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.553+0000 7f9628351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.738 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:36.739 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T21:20:36.750 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-21T21:20:36.005369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80956,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034564,"statfs":{"total":289910292480,"available":289827393536,"internally_reserved":0,"allocated":438272,"data_stored":1256368,"data_compressed":60096,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540537+0000","last_change":"2024-09-21T21:20:34.542593+0000","last_active":"2024-09-21T21:20:34.540537+0000","last_peered":"2024-09-21T21:20:34.540537+0000","last_clean":"2024-09-21T21:20:34.540537+0000","last_became_active":"2024-09-21T21:20:32.619705+0000","last_became_peered":"2024-09-21T21:20:32.619705+0000","last_unstale":"2024-09-21T21:20:34.540537+0000","last_undegraded":"2024-09-21T21:20:34.540537+0000","last_fullsized":"2024-09-21T21:20:34.540537+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.537440+0000","last_change":"2024-09-21T21:20:34.541788+0000","last_active":"2024-09-21T21:20:34.537440+0000","last_peered":"2024-09-21T21:20:34.537440+0000","last_clean":"2024-09-21T21:20:34.537440+0000","last_became_active":"2024-09-21T21:20:32.579460+0000","last_became_peered":"2024-09-21T21:20:32.579460+0000","last_unstale":"2024-09-21T21:20:34.537440+0000","last_undegraded":"2024-09-21T21:20:34.537440+0000","last_fullsized":"2024-09-21T21:20:34.537440+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.547767+0000","last_change":"2024-09-21T21:20:34.548763+0000","last_active":"2024-09-21T21:20:34.547767+0000","last_peered":"2024-09-21T21:20:34.547767+0000","last_clean":"2024-09-21T21:20:34.547767+0000","last_became_active":"2024-09-21T21:20:32.626518+0000","last_became_peered":"2024-09-21T21:20:32.626518+0000","last_unstale":"2024-09-21T21:20:34.547767+0000","last_undegraded":"2024-09-21T21:20:34.547767+0000","last_fullsized":"2024-09-21T21:20:34.547767+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540342+0000","last_change":"2024-09-21T21:20:34.542340+0000","last_active":"2024-09-21T21:20:34.540342+0000","last_peered":"2024-09-21T21:20:34.540342+0000","last_clean":"2024-09-21T21:20:34.540342+0000","last_became_active":"2024-09-21T21:20:32.650357+0000","last_became_peered":"2024-09-21T21:20:32.650357+0000","last_unstale":"2024-09-21T21:20:34.540342+0000","last_undegraded":"2024-09-21T21:20:34.540342+0000","last_fullsized":"2024-09-21T21:20:34.540342+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'52","reported_seq":81,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.545832+0000","last_change":"2024-09-21T21:20:32.600284+0000","last_active":"2024-09-21T21:20:34.545832+0000","last_peered":"2024-09-21T21:20:34.545832+0000","last_clean":"2024-09-21T21:20:34.545832+0000","last_became_active":"2024-09-21T21:20:32.582637+0000","last_became_peered":"2024-09-21T21:20:32.582637+0000","last_unstale":"2024-09-21T21:20:34.545832+0000","last_undegraded":"2024-09-21T21:20:34.545832+0000","last_fullsized":"2024-09-21T21:20:34.545832+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540428+0000","last_change":"2024-09-21T21:20:34.542361+0000","last_active":"2024-09-21T21:20:34.540428+0000","last_peered":"2024-09-21T21:20:34.540428+0000","last_clean":"2024-09-21T21:20:34.540428+0000","last_became_active":"2024-09-21T21:20:32.610632+0000","last_became_peered":"2024-09-21T21:20:32.610632+0000","last_unstale":"2024-09-21T21:20:34.540428+0000","last_undegraded":"2024-09-21T21:20:34.540428+0000","last_fullsized":"2024-09-21T21:20:34.540428+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.553118+0000","last_change":"2024-09-21T21:20:34.553118+0000","last_active":"2024-09-21T21:20:34.553118+0000","last_peered":"2024-09-21T21:20:34.553118+0000","last_clean":"2024-09-21T21:20:34.553118+0000","last_became_active":"2024-09-21T21:20:32.585590+0000","last_became_peered":"2024-09-21T21:20:32.585590+0000","last_unstale":"2024-09-21T21:20:34.553118+0000","last_undegraded":"2024-09-21T21:20:34.553118+0000","last_fullsized":"2024-09-21T21:20:34.553118+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.546280+0000","last_change":"2024-09-21T21:20:34.548538+0000","last_active":"2024-09-21T21:20:34.546280+0000","last_peered":"2024-09-21T21:20:34.546280+0000","last_clean":"2024-09-21T21:20:34.546280+0000","last_became_active":"2024-09-21T21:20:32.610740+0000","last_became_peered":"2024-09-21T21:20:32.610740+0000","last_unstale":"2024-09-21T21:20:34.546280+0000","last_undegraded":"2024-09-21T21:20:34.546280+0000","last_fullsized":"2024-09-21T21:20:34.546280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.537795+0000","last_change":"2024-09-21T21:20:34.541704+0000","last_active":"2024-09-21T21:20:34.537795+0000","last_peered":"2024-09-21T21:20:34.537795+0000","last_clean":"2024-09-21T21:20:34.537795+0000","last_became_active":"2024-09-21T21:20:32.582630+0000","last_became_peered":"2024-09-21T21:20:32.582630+0000","last_unstale":"2024-09-21T21:20:34.537795+0000","last_undegraded":"2024-09-21T21:20:34.537795+0000","last_fullsized":"2024-09-21T21:20:34.537795+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":163840,"data_stored":615566,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":167936,"data_stored":615585,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26936,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344904,"statfs":{"total":96636764160,"available":96609181696,"internally_reserved":0,"allocated":106496,"data_stored":25217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:20:36.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T21:20:36.862 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.857+0000 7f4b2f791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:36.863 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:36.862+0000 7f4b2f791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:37.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:37.047 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-09-21T21:20:37.059 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-21T21:20:36.005369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80956,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034564,"statfs":{"total":289910292480,"available":289827393536,"internally_reserved":0,"allocated":438272,"data_stored":1256368,"data_compressed":60096,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540537+0000","last_change":"2024-09-21T21:20:34.542593+0000","last_active":"2024-09-21T21:20:34.540537+0000","last_peered":"2024-09-21T21:20:34.540537+0000","last_clean":"2024-09-21T21:20:34.540537+0000","last_became_active":"2024-09-21T21:20:32.619705+0000","last_became_peered":"2024-09-21T21:20:32.619705+0000","last_unstale":"2024-09-21T21:20:34.540537+0000","last_undegraded":"2024-09-21T21:20:34.540537+0000","last_fullsized":"2024-09-21T21:20:34.540537+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.537440+0000","last_change":"2024-09-21T21:20:34.541788+0000","last_active":"2024-09-21T21:20:34.537440+0000","last_peered":"2024-09-21T21:20:34.537440+0000","last_clean":"2024-09-21T21:20:34.537440+0000","last_became_active":"2024-09-21T21:20:32.579460+0000","last_became_peered":"2024-09-21T21:20:32.579460+0000","last_unstale":"2024-09-21T21:20:34.537440+0000","last_undegraded":"2024-09-21T21:20:34.537440+0000","last_fullsized":"2024-09-21T21:20:34.537440+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.547767+0000","last_change":"2024-09-21T21:20:34.548763+0000","last_active":"2024-09-21T21:20:34.547767+0000","last_peered":"2024-09-21T21:20:34.547767+0000","last_clean":"2024-09-21T21:20:34.547767+0000","last_became_active":"2024-09-21T21:20:32.626518+0000","last_became_peered":"2024-09-21T21:20:32.626518+0000","last_unstale":"2024-09-21T21:20:34.547767+0000","last_undegraded":"2024-09-21T21:20:34.547767+0000","last_fullsized":"2024-09-21T21:20:34.547767+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540342+0000","last_change":"2024-09-21T21:20:34.542340+0000","last_active":"2024-09-21T21:20:34.540342+0000","last_peered":"2024-09-21T21:20:34.540342+0000","last_clean":"2024-09-21T21:20:34.540342+0000","last_became_active":"2024-09-21T21:20:32.650357+0000","last_became_peered":"2024-09-21T21:20:32.650357+0000","last_unstale":"2024-09-21T21:20:34.540342+0000","last_undegraded":"2024-09-21T21:20:34.540342+0000","last_fullsized":"2024-09-21T21:20:34.540342+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"10'52","reported_seq":81,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.545832+0000","last_change":"2024-09-21T21:20:32.600284+0000","last_active":"2024-09-21T21:20:34.545832+0000","last_peered":"2024-09-21T21:20:34.545832+0000","last_clean":"2024-09-21T21:20:34.545832+0000","last_became_active":"2024-09-21T21:20:32.582637+0000","last_became_peered":"2024-09-21T21:20:32.582637+0000","last_unstale":"2024-09-21T21:20:34.545832+0000","last_undegraded":"2024-09-21T21:20:34.545832+0000","last_fullsized":"2024-09-21T21:20:34.545832+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"10'1","reported_seq":16,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.540428+0000","last_change":"2024-09-21T21:20:34.542361+0000","last_active":"2024-09-21T21:20:34.540428+0000","last_peered":"2024-09-21T21:20:34.540428+0000","last_clean":"2024-09-21T21:20:34.540428+0000","last_became_active":"2024-09-21T21:20:32.610632+0000","last_became_peered":"2024-09-21T21:20:32.610632+0000","last_unstale":"2024-09-21T21:20:34.540428+0000","last_undegraded":"2024-09-21T21:20:34.540428+0000","last_fullsized":"2024-09-21T21:20:34.540428+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"12'2","reported_seq":17,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.553118+0000","last_change":"2024-09-21T21:20:34.553118+0000","last_active":"2024-09-21T21:20:34.553118+0000","last_peered":"2024-09-21T21:20:34.553118+0000","last_clean":"2024-09-21T21:20:34.553118+0000","last_became_active":"2024-09-21T21:20:32.585590+0000","last_became_peered":"2024-09-21T21:20:32.585590+0000","last_unstale":"2024-09-21T21:20:34.553118+0000","last_undegraded":"2024-09-21T21:20:34.553118+0000","last_fullsized":"2024-09-21T21:20:34.553118+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.546280+0000","last_change":"2024-09-21T21:20:34.548538+0000","last_active":"2024-09-21T21:20:34.546280+0000","last_peered":"2024-09-21T21:20:34.546280+0000","last_clean":"2024-09-21T21:20:34.546280+0000","last_became_active":"2024-09-21T21:20:32.610740+0000","last_became_peered":"2024-09-21T21:20:32.610740+0000","last_unstale":"2024-09-21T21:20:34.546280+0000","last_undegraded":"2024-09-21T21:20:34.546280+0000","last_fullsized":"2024-09-21T21:20:34.546280+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-09-21T21:20:34.537795+0000","last_change":"2024-09-21T21:20:34.541704+0000","last_active":"2024-09-21T21:20:34.537795+0000","last_peered":"2024-09-21T21:20:34.537795+0000","last_clean":"2024-09-21T21:20:34.537795+0000","last_became_active":"2024-09-21T21:20:32.582630+0000","last_became_peered":"2024-09-21T21:20:32.582630+0000","last_unstale":"2024-09-21T21:20:34.537795+0000","last_undegraded":"2024-09-21T21:20:34.537795+0000","last_fullsized":"2024-09-21T21:20:34.537795+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T21:20:31.509978+0000","last_clean_scrub_stamp":"2024-09-21T21:20:31.509978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27008,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344832,"statfs":{"total":96636764160,"available":96609107968,"internally_reserved":0,"allocated":163840,"data_stored":615566,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":164,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":167936,"data_stored":615585,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26936,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344904,"statfs":{"total":96636764160,"available":96609181696,"internally_reserved":0,"allocated":106496,"data_stored":25217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":590368,"data_compressed":30048,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T21:20:37.060 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T21:20:37.060 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T21:20:37.060 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T21:20:37.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T21:20:37.169 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:37.165+0000 7fd490d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:37.171 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21T21:20:37.170+0000 7fd490d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-21T21:20:37.395 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:37.395 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T21:20:37.408 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T21:20:37.408 INFO:teuthology.run_tasks:Running task cbt... 2024-09-21T21:20:37.425 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi107.front.sepia.ceph.com', 'osds': ['ubuntu@smithi107.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi107.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi107.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi107.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-21T21:20:37.426 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-21T21:20:37.444 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-21T21:20:37.444 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:20:37.510 DEBUG:teuthology.orchestra.run.smithi107:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-21T21:20:37.573 INFO:teuthology.orchestra.run.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-21T21:20:38.030 INFO:tasks.cbt:Installing collectl 2024-09-21T21:20:38.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-21T21:20:38.126 INFO:teuthology.orchestra.run.smithi107.stderr:--2024-09-21 21:20:38-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-21T21:20:38.270 INFO:teuthology.orchestra.run.smithi107.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-09-21T21:20:38.278 INFO:teuthology.orchestra.run.smithi107.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-09-21T21:20:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:20:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zimcJRcxs94N1wQyDOCQ1dyWl2np7voaZwe-2w5ngF0i2jBKrVKmWOrNVuFFfLymZRiR9BTt43mZibjUp9Oy2SfeQ%3D%3D&use_mirror=psychz&r= [following] 2024-09-21T21:20:38.510 INFO:teuthology.orchestra.run.smithi107.stderr:--2024-09-21 21:20:38-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm7zimcJRcxs94N1wQyDOCQ1dyWl2np7voaZwe-2w5ngF0i2jBKrVKmWOrNVuFFfLymZRiR9BTt43mZibjUp9Oy2SfeQ%3D%3D&use_mirror=psychz&r= 2024-09-21T21:20:38.511 INFO:teuthology.orchestra.run.smithi107.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-21T21:20:38.584 INFO:teuthology.orchestra.run.smithi107.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-21T21:20:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-21T21:20:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:Location: https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-21T21:20:38.897 INFO:teuthology.orchestra.run.smithi107.stderr:--2024-09-21 21:20:38-- https://psychz.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-21T21:20:38.908 INFO:teuthology.orchestra.run.smithi107.stderr:Resolving psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)... 208.87.241.191 2024-09-21T21:20:38.969 INFO:teuthology.orchestra.run.smithi107.stderr:Connecting to psychz.dl.sourceforge.net (psychz.dl.sourceforge.net)|208.87.241.191|:443... connected. 2024-09-21T21:20:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-21T21:20:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-21T21:20:39.518 INFO:teuthology.orchestra.run.smithi107.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-21T21:20:39.518 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:39.579 INFO:teuthology.orchestra.run.smithi107.stderr: 0K .......... .......... .......... .......... .......... 8% 808K 1s 2024-09-21T21:20:39.701 INFO:teuthology.orchestra.run.smithi107.stderr: 50K .......... .......... .......... .......... .......... 16% 414K 1s 2024-09-21T21:20:39.701 INFO:teuthology.orchestra.run.smithi107.stderr: 100K .......... .......... .......... .......... .......... 24% 71.9M 1s 2024-09-21T21:20:39.740 INFO:teuthology.orchestra.run.smithi107.stderr: 150K .......... .......... .......... .......... .......... 32% 1.26M 0s 2024-09-21T21:20:39.763 INFO:teuthology.orchestra.run.smithi107.stderr: 200K .......... .......... .......... .......... .......... 41% 2.26M 0s 2024-09-21T21:20:39.764 INFO:teuthology.orchestra.run.smithi107.stderr: 250K .......... .......... .......... .......... .......... 49% 116M 0s 2024-09-21T21:20:39.764 INFO:teuthology.orchestra.run.smithi107.stderr: 300K .......... .......... .......... .......... .......... 57% 112M 0s 2024-09-21T21:20:39.801 INFO:teuthology.orchestra.run.smithi107.stderr: 350K .......... .......... .......... .......... .......... 65% 1.28M 0s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 400K .......... .......... .......... .......... .......... 74% 2.27M 0s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 450K .......... .......... .......... .......... .......... 82% 112M 0s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 500K .......... .......... .......... .......... .......... 90% 121M 0s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 550K .......... .......... .......... .......... .......... 98% 112M 0s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 600K ...... 100% 11.4T=0.3s 2024-09-21T21:20:39.824 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:39.825 INFO:teuthology.orchestra.run.smithi107.stderr:2024-09-21 21:20:39 (1.94 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-21T21:20:39.825 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-21T21:20:39.909 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/ 2024-09-21T21:20:39.909 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/collectl.conf 2024-09-21T21:20:39.909 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/ 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Socket.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/colmux.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Import.html 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/style.css 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-21T21:20:39.910 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Process.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-21T21:20:39.911 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Export.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Startup.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Performance.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Messages.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Logging.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-21T21:20:39.912 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Misc.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Examples.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Data.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-21T21:20:39.913 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Playback.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Hello.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Network.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Memory.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/Disks.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-21T21:20:39.914 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/service/ 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/service/collectl.service 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/initd/ 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/initd/collectl 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/misc.ph 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/gexpr.ph 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/statsd.ph 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/envrules.std 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/hello.ph 2024-09-21T21:20:39.915 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/vmsum.ph 2024-09-21T21:20:39.916 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/proctree.ph 2024-09-21T21:20:39.916 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/README-WINDOWS 2024-09-21T21:20:39.916 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/formatit.ph 2024-09-21T21:20:39.916 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/colmux 2024-09-21T21:20:39.917 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/INSTALL 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/graphite.ph 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/vmstat.ph 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/README 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/man1/ 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/man1/collectl.1 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/man1/colmux.1 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/lexpr.ph 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/client.pl 2024-09-21T21:20:39.918 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/collectl 2024-09-21T21:20:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/COPYING 2024-09-21T21:20:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/vnet.ph 2024-09-21T21:20:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/GPL 2024-09-21T21:20:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/UNINSTALL 2024-09-21T21:20:39.919 INFO:teuthology.orchestra.run.smithi107.stdout:collectl-4.3.1/ARTISTIC 2024-09-21T21:20:39.970 INFO:teuthology.orchestra.run.smithi107.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-21T21:20:39.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-21T21:20:40.578 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:10 ago on Sat 21 Sep 2024 09:17:30 PM UTC. 2024-09-21T21:20:40.678 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-21T21:20:40.680 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-21T21:20:40.728 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-21T21:20:40.729 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:20:40.729 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-21T21:20:40.729 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:20:40.729 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout:Install 5 Packages 2024-09-21T21:20:40.730 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:40.731 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.4 M 2024-09-21T21:20:40.731 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 11 M 2024-09-21T21:20:40.731 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-21T21:20:41.356 INFO:teuthology.orchestra.run.smithi107.stdout:(1/5): libradospp-devel-19.3.0-5092.g62da54b4.e 204 kB/s | 31 kB 00:00 2024-09-21T21:20:41.381 INFO:teuthology.orchestra.run.smithi107.stdout:(2/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 135 kB/s | 24 kB 00:00 2024-09-21T21:20:41.424 INFO:teuthology.orchestra.run.smithi107.stdout:(3/5): pdsh-2.34-7.el9.x86_64.rpm 1.7 MB/s | 118 kB 00:00 2024-09-21T21:20:41.449 INFO:teuthology.orchestra.run.smithi107.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 195 kB/s | 13 kB 00:00 2024-09-21T21:20:41.591 INFO:teuthology.orchestra.run.smithi107.stdout:(5/5): perf-5.14.0-508.el9.x86_64.rpm 5.8 MB/s | 2.2 MB 00:00 2024-09-21T21:20:41.594 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-21T21:20:41.595 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.8 MB/s | 2.4 MB 00:00 2024-09-21T21:20:41.625 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-21T21:20:41.646 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-21T21:20:41.646 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-21T21:20:41.785 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-21T21:20:41.785 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-21T21:20:42.144 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-21T21:20:42.210 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-21T21:20:42.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-21T21:20:42.280 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-21T21:20:42.608 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-21T21:20:42.694 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:20:43.616 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-21T21:20:43.616 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-21T21:20:43.616 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-21T21:20:43.616 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-21T21:20:43.616 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: pdsh-2.34-7.el9.x86_64 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: perf-5.14.0-508.el9.x86_64 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:43.943 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-21T21:20:44.351 INFO:tasks.cbt:benchmark: librbdfio 2024-09-21T21:20:44.352 DEBUG:teuthology.orchestra.run.smithi107:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-09-21T21:20:44.366 INFO:teuthology.orchestra.run.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-09-21T21:20:46.546 DEBUG:teuthology.orchestra.run.smithi107:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:Operating system Linux 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:CPU x86_64 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:Big endian no 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:Compiler gcc 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:Cross compile no 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-21T21:20:47.119 INFO:teuthology.orchestra.run.smithi107.stdout:Static build no 2024-09-21T21:20:47.231 INFO:teuthology.orchestra.run.smithi107.stdout:Wordsize 64 2024-09-21T21:20:47.239 INFO:teuthology.orchestra.run.smithi107.stdout:zlib no 2024-09-21T21:20:47.302 INFO:teuthology.orchestra.run.smithi107.stdout:fcntl(F_FULLFSYNC) no 2024-09-21T21:20:47.317 INFO:teuthology.orchestra.run.smithi107.stdout:Linux AIO support no 2024-09-21T21:20:47.317 INFO:teuthology.orchestra.run.smithi107.stdout:Linux AIO support rw flags no 2024-09-21T21:20:47.384 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX AIO support yes 2024-09-21T21:20:47.384 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX AIO support needs -lrt no 2024-09-21T21:20:47.416 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX AIO fsync yes 2024-09-21T21:20:47.446 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX pshared support yes 2024-09-21T21:20:47.495 INFO:teuthology.orchestra.run.smithi107.stdout:pthread_condattr_setclock() yes 2024-09-21T21:20:47.543 INFO:teuthology.orchestra.run.smithi107.stdout:pthread_sigmask() yes 2024-09-21T21:20:47.579 INFO:teuthology.orchestra.run.smithi107.stdout:pthread_getaffinity_np() yes 2024-09-21T21:20:47.589 INFO:teuthology.orchestra.run.smithi107.stdout:Solaris AIO support no 2024-09-21T21:20:47.634 INFO:teuthology.orchestra.run.smithi107.stdout:__sync_fetch_and_add yes 2024-09-21T21:20:47.674 INFO:teuthology.orchestra.run.smithi107.stdout:__sync_synchronize yes 2024-09-21T21:20:47.704 INFO:teuthology.orchestra.run.smithi107.stdout:__sync_val_compare_and_swap yes 2024-09-21T21:20:47.712 INFO:teuthology.orchestra.run.smithi107.stdout:libverbs no 2024-09-21T21:20:47.733 INFO:teuthology.orchestra.run.smithi107.stdout:rdmacm no 2024-09-21T21:20:47.764 INFO:teuthology.orchestra.run.smithi107.stdout:asprintf() yes 2024-09-21T21:20:47.794 INFO:teuthology.orchestra.run.smithi107.stdout:vasprintf() yes 2024-09-21T21:20:47.825 INFO:teuthology.orchestra.run.smithi107.stdout:Linux fallocate yes 2024-09-21T21:20:47.857 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX fadvise yes 2024-09-21T21:20:47.890 INFO:teuthology.orchestra.run.smithi107.stdout:POSIX fallocate yes 2024-09-21T21:20:47.920 INFO:teuthology.orchestra.run.smithi107.stdout:sched_setaffinity(3 arg) yes 2024-09-21T21:20:47.920 INFO:teuthology.orchestra.run.smithi107.stdout:sched_setaffinity(2 arg) no 2024-09-21T21:20:47.951 INFO:teuthology.orchestra.run.smithi107.stdout:clock_gettime yes 2024-09-21T21:20:47.983 INFO:teuthology.orchestra.run.smithi107.stdout:CLOCK_MONOTONIC yes 2024-09-21T21:20:48.015 INFO:teuthology.orchestra.run.smithi107.stdout:clockid_t yes 2024-09-21T21:20:48.056 INFO:teuthology.orchestra.run.smithi107.stdout:gettimeofday yes 2024-09-21T21:20:48.088 INFO:teuthology.orchestra.run.smithi107.stdout:fdatasync yes 2024-09-21T21:20:48.119 INFO:teuthology.orchestra.run.smithi107.stdout:pipe() yes 2024-09-21T21:20:48.151 INFO:teuthology.orchestra.run.smithi107.stdout:pipe2() yes 2024-09-21T21:20:48.182 INFO:teuthology.orchestra.run.smithi107.stdout:pread() yes 2024-09-21T21:20:48.254 INFO:teuthology.orchestra.run.smithi107.stdout:sync_file_range yes 2024-09-21T21:20:48.262 INFO:teuthology.orchestra.run.smithi107.stdout:ASharedMemory_create no 2024-09-21T21:20:48.274 INFO:teuthology.orchestra.run.smithi107.stdout:EXT4 move extent yes 2024-09-21T21:20:48.306 INFO:teuthology.orchestra.run.smithi107.stdout:Linux splice(2) yes 2024-09-21T21:20:48.313 INFO:teuthology.orchestra.run.smithi107.stdout:libnuma no 2024-09-21T21:20:48.342 INFO:teuthology.orchestra.run.smithi107.stdout:strsep yes 2024-09-21T21:20:48.373 INFO:teuthology.orchestra.run.smithi107.stdout:strcasestr yes 2024-09-21T21:20:48.388 INFO:teuthology.orchestra.run.smithi107.stdout:strlcat no 2024-09-21T21:20:48.422 INFO:teuthology.orchestra.run.smithi107.stdout:getopt_long_only() yes 2024-09-21T21:20:48.488 INFO:teuthology.orchestra.run.smithi107.stdout:inet_aton yes 2024-09-21T21:20:48.519 INFO:teuthology.orchestra.run.smithi107.stdout:socklen_t yes 2024-09-21T21:20:48.552 INFO:teuthology.orchestra.run.smithi107.stdout:__thread yes 2024-09-21T21:20:48.597 INFO:teuthology.orchestra.run.smithi107.stdout:RUSAGE_THREAD yes 2024-09-21T21:20:48.635 INFO:teuthology.orchestra.run.smithi107.stdout:SCHED_IDLE yes 2024-09-21T21:20:48.671 INFO:teuthology.orchestra.run.smithi107.stdout:TCP_NODELAY yes 2024-09-21T21:20:48.704 INFO:teuthology.orchestra.run.smithi107.stdout:vsock yes 2024-09-21T21:20:48.737 INFO:teuthology.orchestra.run.smithi107.stdout:Net engine window_size yes 2024-09-21T21:20:48.773 INFO:teuthology.orchestra.run.smithi107.stdout:TCP_MAXSEG yes 2024-09-21T21:20:48.803 INFO:teuthology.orchestra.run.smithi107.stdout:RLIMIT_MEMLOCK yes 2024-09-21T21:20:48.837 INFO:teuthology.orchestra.run.smithi107.stdout:pwritev/preadv yes 2024-09-21T21:20:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:pwritev2/preadv2 yes 2024-09-21T21:20:48.907 INFO:teuthology.orchestra.run.smithi107.stdout:IPv6 helpers yes 2024-09-21T21:20:48.923 INFO:teuthology.orchestra.run.smithi107.stdout:http engine no 2024-09-21T21:20:49.067 INFO:teuthology.orchestra.run.smithi107.stdout:Rados engine yes 2024-09-21T21:20:49.257 INFO:teuthology.orchestra.run.smithi107.stdout:Rados Block Device engine yes 2024-09-21T21:20:49.446 INFO:teuthology.orchestra.run.smithi107.stdout:rbd_poll yes 2024-09-21T21:20:49.633 INFO:teuthology.orchestra.run.smithi107.stdout:rbd_invalidate_cache yes 2024-09-21T21:20:49.664 INFO:teuthology.orchestra.run.smithi107.stdout:setvbuf yes 2024-09-21T21:20:49.672 INFO:teuthology.orchestra.run.smithi107.stdout:Gluster API engine no 2024-09-21T21:20:49.688 INFO:teuthology.orchestra.run.smithi107.stdout:s390_z196_facilities no 2024-09-21T21:20:49.688 INFO:teuthology.orchestra.run.smithi107.stdout:HDFS engine no 2024-09-21T21:20:49.736 INFO:teuthology.orchestra.run.smithi107.stdout:MTD yes 2024-09-21T21:20:49.743 INFO:teuthology.orchestra.run.smithi107.stdout:libpmem no 2024-09-21T21:20:49.743 INFO:teuthology.orchestra.run.smithi107.stdout:libpmem1_5 no 2024-09-21T21:20:49.751 INFO:teuthology.orchestra.run.smithi107.stdout:libpmem2 no 2024-09-21T21:20:49.752 INFO:teuthology.orchestra.run.smithi107.stdout:PMDK dev-dax engine no 2024-09-21T21:20:49.752 INFO:teuthology.orchestra.run.smithi107.stdout:PMDK libpmem engine no 2024-09-21T21:20:49.759 INFO:teuthology.orchestra.run.smithi107.stdout:DDN's Infinite Memory Engine no 2024-09-21T21:20:49.759 INFO:teuthology.orchestra.run.smithi107.stdout:iscsi engine no 2024-09-21T21:20:49.760 INFO:teuthology.orchestra.run.smithi107.stdout:NBD engine no 2024-09-21T21:20:49.769 INFO:teuthology.orchestra.run.smithi107.stdout:DAOS File System (dfs) Engine no 2024-09-21T21:20:49.777 INFO:teuthology.orchestra.run.smithi107.stdout:NFS engine no 2024-09-21T21:20:49.777 INFO:teuthology.orchestra.run.smithi107.stdout:lex/yacc for arithmetic no 2024-09-21T21:20:49.816 INFO:teuthology.orchestra.run.smithi107.stdout:getmntent yes 2024-09-21T21:20:49.851 INFO:teuthology.orchestra.run.smithi107.stdout:getmntinfo no 2024-09-21T21:20:49.911 INFO:teuthology.orchestra.run.smithi107.stdout:Static Assert yes 2024-09-21T21:20:49.938 INFO:teuthology.orchestra.run.smithi107.stdout:bool yes 2024-09-21T21:20:49.971 INFO:teuthology.orchestra.run.smithi107.stdout:strndup yes 2024-09-21T21:20:49.979 INFO:teuthology.orchestra.run.smithi107.stdout:Valgrind headers no 2024-09-21T21:20:50.008 INFO:teuthology.orchestra.run.smithi107.stdout:Zoned block device support yes 2024-09-21T21:20:50.036 INFO:teuthology.orchestra.run.smithi107.stdout:Zoned block device capacity yes 2024-09-21T21:20:50.044 INFO:teuthology.orchestra.run.smithi107.stdout:libzbc engine no 2024-09-21T21:20:50.072 INFO:teuthology.orchestra.run.smithi107.stdout:NVMe uring command support yes 2024-09-21T21:20:50.087 INFO:teuthology.orchestra.run.smithi107.stdout:xnvme engine no 2024-09-21T21:20:50.095 INFO:teuthology.orchestra.run.smithi107.stdout:isal no 2024-09-21T21:20:50.110 INFO:teuthology.orchestra.run.smithi107.stdout:libblkio engine no 2024-09-21T21:20:50.110 INFO:teuthology.orchestra.run.smithi107.stdout:march_armv8_a_crc_crypto no 2024-09-21T21:20:50.111 INFO:teuthology.orchestra.run.smithi107.stdout:cuda no 2024-09-21T21:20:50.111 INFO:teuthology.orchestra.run.smithi107.stdout:libcufile no 2024-09-21T21:20:50.139 INFO:teuthology.orchestra.run.smithi107.stdout:Build march=native yes 2024-09-21T21:20:50.146 INFO:teuthology.orchestra.run.smithi107.stdout:CUnit no 2024-09-21T21:20:50.176 INFO:teuthology.orchestra.run.smithi107.stdout:__kernel_rwf_t yes 2024-09-21T21:20:50.203 INFO:teuthology.orchestra.run.smithi107.stdout:-Wimplicit-fallthrough=2 yes 2024-09-21T21:20:50.234 INFO:teuthology.orchestra.run.smithi107.stdout:-Wno-stringop-truncation yes 2024-09-21T21:20:50.271 INFO:teuthology.orchestra.run.smithi107.stdout:MADV_HUGEPAGE yes 2024-09-21T21:20:50.301 INFO:teuthology.orchestra.run.smithi107.stdout:gettid yes 2024-09-21T21:20:50.337 INFO:teuthology.orchestra.run.smithi107.stdout:statx(2)/libc yes 2024-09-21T21:20:50.371 INFO:teuthology.orchestra.run.smithi107.stdout:statx(2)/syscall yes 2024-09-21T21:20:50.373 INFO:teuthology.orchestra.run.smithi107.stdout:Windows PDB generation no 2024-09-21T21:20:50.404 INFO:teuthology.orchestra.run.smithi107.stdout:timerfd_create yes 2024-09-21T21:20:50.406 INFO:teuthology.orchestra.run.smithi107.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-09-21T21:20:50.406 INFO:teuthology.orchestra.run.smithi107.stdout:Lib-based ioengines dynamic no 2024-09-21T21:20:50.468 INFO:teuthology.orchestra.run.smithi107.stdout:TCMalloc support yes 2024-09-21T21:20:50.469 INFO:teuthology.orchestra.run.smithi107.stdout:seed_buckets 4 2024-09-21T21:20:50.545 INFO:teuthology.orchestra.run.smithi107.stderr:FIO_VERSION = fio-3.37-124-gbcd4 2024-09-21T21:20:50.581 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc16.o 2024-09-21T21:20:50.674 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc32.o 2024-09-21T21:20:50.711 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc32c-arm64.o 2024-09-21T21:20:50.801 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc32c-intel.o 2024-09-21T21:20:50.847 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc32c.o 2024-09-21T21:20:50.887 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc64.o 2024-09-21T21:20:50.927 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crc7.o 2024-09-21T21:20:50.961 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/crct10dif_common.o 2024-09-21T21:20:51.001 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/fnv.o 2024-09-21T21:20:51.042 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/md5.o 2024-09-21T21:20:51.144 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/murmur3.o 2024-09-21T21:20:51.188 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/sha1.o 2024-09-21T21:20:51.377 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/sha256.o 2024-09-21T21:20:51.665 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/sha3.o 2024-09-21T21:20:51.812 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/sha512.o 2024-09-21T21:20:51.933 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/test.o 2024-09-21T21:20:52.100 INFO:teuthology.orchestra.run.smithi107.stdout: CC crc/xxhash.o 2024-09-21T21:20:52.200 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/axmap.o 2024-09-21T21:20:52.325 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/bloom.o 2024-09-21T21:20:52.412 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/flist_sort.o 2024-09-21T21:20:52.487 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/gauss.o 2024-09-21T21:20:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/getrusage.o 2024-09-21T21:20:52.623 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/hweight.o 2024-09-21T21:20:52.663 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/ieee754.o 2024-09-21T21:20:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/lfsr.o 2024-09-21T21:20:52.814 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/memalign.o 2024-09-21T21:20:52.859 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/memcpy.o 2024-09-21T21:20:53.003 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/mountcheck.o 2024-09-21T21:20:53.045 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/num2str.o 2024-09-21T21:20:53.111 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/output_buffer.o 2024-09-21T21:20:53.159 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/pattern.o 2024-09-21T21:20:53.309 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/prio_tree.o 2024-09-21T21:20:53.429 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/rand.o 2024-09-21T21:20:53.620 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/rbtree.o 2024-09-21T21:20:53.744 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/strntol.o 2024-09-21T21:20:53.792 INFO:teuthology.orchestra.run.smithi107.stdout: CC lib/zipf.o 2024-09-21T21:20:53.913 INFO:teuthology.orchestra.run.smithi107.stdout: CC gettime.o 2024-09-21T21:20:54.207 INFO:teuthology.orchestra.run.smithi107.stdout: CC ioengines.o 2024-09-21T21:20:54.473 INFO:teuthology.orchestra.run.smithi107.stdout: CC init.o 2024-09-21T21:20:55.241 INFO:teuthology.orchestra.run.smithi107.stdout: CC stat.o 2024-09-21T21:20:57.093 INFO:teuthology.orchestra.run.smithi107.stdout: CC log.o 2024-09-21T21:20:57.220 INFO:teuthology.orchestra.run.smithi107.stdout: CC time.o 2024-09-21T21:20:57.347 INFO:teuthology.orchestra.run.smithi107.stdout: CC filesetup.o 2024-09-21T21:20:58.046 INFO:teuthology.orchestra.run.smithi107.stdout: CC eta.o 2024-09-21T21:20:58.369 INFO:teuthology.orchestra.run.smithi107.stdout: CC verify.o 2024-09-21T21:20:59.050 INFO:teuthology.orchestra.run.smithi107.stdout: CC memory.o 2024-09-21T21:20:59.225 INFO:teuthology.orchestra.run.smithi107.stdout: CC io_u.o 2024-09-21T21:21:00.014 INFO:teuthology.orchestra.run.smithi107.stdout: CC parse.o 2024-09-21T21:21:00.458 INFO:teuthology.orchestra.run.smithi107.stdout: CC fio_sem.o 2024-09-21T21:21:00.565 INFO:teuthology.orchestra.run.smithi107.stdout: CC rwlock.o 2024-09-21T21:21:00.649 INFO:teuthology.orchestra.run.smithi107.stdout: CC pshared.o 2024-09-21T21:21:00.717 INFO:teuthology.orchestra.run.smithi107.stdout: CC options.o 2024-09-21T21:21:01.520 INFO:teuthology.orchestra.run.smithi107.stdout: CC smalloc.o 2024-09-21T21:21:01.706 INFO:teuthology.orchestra.run.smithi107.stdout: CC filehash.o 2024-09-21T21:21:01.856 INFO:teuthology.orchestra.run.smithi107.stdout: CC profile.o 2024-09-21T21:21:01.977 INFO:teuthology.orchestra.run.smithi107.stdout: CC debug.o 2024-09-21T21:21:02.019 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/cpu.o 2024-09-21T21:21:02.156 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/mmap.o 2024-09-21T21:21:02.325 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/sync.o 2024-09-21T21:21:02.551 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/null.o 2024-09-21T21:21:02.667 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/net.o 2024-09-21T21:21:03.101 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/ftruncate.o 2024-09-21T21:21:03.201 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/fileoperations.o 2024-09-21T21:21:03.358 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/exec.o 2024-09-21T21:21:03.508 INFO:teuthology.orchestra.run.smithi107.stdout: CC server.o 2024-09-21T21:21:04.264 INFO:teuthology.orchestra.run.smithi107.stdout: CC client.o 2024-09-21T21:21:04.833 INFO:teuthology.orchestra.run.smithi107.stdout: CC iolog.o 2024-09-21T21:21:05.292 INFO:teuthology.orchestra.run.smithi107.stdout: CC backend.o 2024-09-21T21:21:06.146 INFO:teuthology.orchestra.run.smithi107.stdout: CC libfio.o 2024-09-21T21:21:06.322 INFO:teuthology.orchestra.run.smithi107.stdout: CC flow.o 2024-09-21T21:21:06.436 INFO:teuthology.orchestra.run.smithi107.stdout: CC cconv.o 2024-09-21T21:21:06.795 INFO:teuthology.orchestra.run.smithi107.stdout: CC gettime-thread.o 2024-09-21T21:21:06.897 INFO:teuthology.orchestra.run.smithi107.stdout: CC helpers.o 2024-09-21T21:21:06.960 INFO:teuthology.orchestra.run.smithi107.stdout: CC json.o 2024-09-21T21:21:07.143 INFO:teuthology.orchestra.run.smithi107.stdout: CC idletime.o 2024-09-21T21:21:07.370 INFO:teuthology.orchestra.run.smithi107.stdout: CC td_error.o 2024-09-21T21:21:07.471 INFO:teuthology.orchestra.run.smithi107.stdout: CC profiles/tiobench.o 2024-09-21T21:21:07.570 INFO:teuthology.orchestra.run.smithi107.stdout: CC profiles/act.o 2024-09-21T21:21:07.729 INFO:teuthology.orchestra.run.smithi107.stdout: CC io_u_queue.o 2024-09-21T21:21:07.780 INFO:teuthology.orchestra.run.smithi107.stdout: CC filelock.o 2024-09-21T21:21:07.907 INFO:teuthology.orchestra.run.smithi107.stdout: CC workqueue.o 2024-09-21T21:21:08.087 INFO:teuthology.orchestra.run.smithi107.stdout: CC rate-submit.o 2024-09-21T21:21:08.239 INFO:teuthology.orchestra.run.smithi107.stdout: CC optgroup.o 2024-09-21T21:21:08.287 INFO:teuthology.orchestra.run.smithi107.stdout: CC helper_thread.o 2024-09-21T21:21:08.467 INFO:teuthology.orchestra.run.smithi107.stdout: CC steadystate.o 2024-09-21T21:21:08.704 INFO:teuthology.orchestra.run.smithi107.stdout: CC zone-dist.o 2024-09-21T21:21:08.840 INFO:teuthology.orchestra.run.smithi107.stdout: CC zbd.o 2024-09-21T21:21:09.535 INFO:teuthology.orchestra.run.smithi107.stdout: CC dedupe.o 2024-09-21T21:21:09.686 INFO:teuthology.orchestra.run.smithi107.stdout: CC dataplacement.o 2024-09-21T21:21:09.857 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/posixaio.o 2024-09-21T21:21:09.991 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/falloc.o 2024-09-21T21:21:10.102 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/e4defrag.o 2024-09-21T21:21:10.233 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/splice.o 2024-09-21T21:21:10.422 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/asprintf.o 2024-09-21T21:21:10.459 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/strlcat.o 2024-09-21T21:21:10.503 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/mtd.o 2024-09-21T21:21:10.668 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/libmtd.o 2024-09-21T21:21:11.049 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/libmtd_legacy.o 2024-09-21T21:21:11.181 INFO:teuthology.orchestra.run.smithi107.stdout: CC diskutil.o 2024-09-21T21:21:11.392 INFO:teuthology.orchestra.run.smithi107.stdout: CC fifo.o 2024-09-21T21:21:11.449 INFO:teuthology.orchestra.run.smithi107.stdout: CC blktrace.o 2024-09-21T21:21:11.780 INFO:teuthology.orchestra.run.smithi107.stdout: CC cgroup.o 2024-09-21T21:21:11.928 INFO:teuthology.orchestra.run.smithi107.stdout: CC trim.o 2024-09-21T21:21:12.044 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/sg.o 2024-09-21T21:21:12.425 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/linux-dev-lookup.o 2024-09-21T21:21:12.484 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/io_uring.o 2024-09-21T21:21:12.901 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/nvme.o 2024-09-21T21:21:13.215 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/linux-blkzoned.o 2024-09-21T21:21:13.368 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/cmdprio.o 2024-09-21T21:21:13.611 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/rados.o 2024-09-21T21:21:13.787 INFO:teuthology.orchestra.run.smithi107.stdout: CC engines/rbd.o 2024-09-21T21:21:13.970 INFO:teuthology.orchestra.run.smithi107.stdout: CC fio.o 2024-09-21T21:21:14.060 INFO:teuthology.orchestra.run.smithi107.stdout: LINK fio 2024-09-21T21:21:14.334 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/log.o 2024-09-21T21:21:14.377 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/genzipf.o 2024-09-21T21:21:14.517 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/strcasestr.o 2024-09-21T21:21:14.543 INFO:teuthology.orchestra.run.smithi107.stdout: CC oslib/strndup.o 2024-09-21T21:21:14.567 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/fio-genzipf 2024-09-21T21:21:14.742 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/btrace2fio.o 2024-09-21T21:21:15.163 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/fio-btrace2fio 2024-09-21T21:21:15.334 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/verify-state.o 2024-09-21T21:21:15.432 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/debug.o 2024-09-21T21:21:15.468 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/fio-verify-state 2024-09-21T21:21:15.641 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/stest.o 2024-09-21T21:21:15.717 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/arch.o 2024-09-21T21:21:15.748 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/stest 2024-09-21T21:21:15.922 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/ieee754.o 2024-09-21T21:21:15.963 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/ieee754 2024-09-21T21:21:16.136 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/axmap.o 2024-09-21T21:21:16.236 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/axmap 2024-09-21T21:21:16.411 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/lfsr-test.o 2024-09-21T21:21:16.510 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/lfsr-test 2024-09-21T21:21:16.686 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/gen-rand.o 2024-09-21T21:21:16.784 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/gen-rand 2024-09-21T21:21:16.958 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/memlock.o 2024-09-21T21:21:17.020 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/memlock 2024-09-21T21:21:17.190 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/read-to-pipe-async.o 2024-09-21T21:21:17.382 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/read-to-pipe-async 2024-09-21T21:21:17.553 INFO:teuthology.orchestra.run.smithi107.stdout: CC t/io_uring.o 2024-09-21T21:21:18.091 INFO:teuthology.orchestra.run.smithi107.stdout: LINK t/io_uring 2024-09-21T21:21:18.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-21T21:21:18.297 DEBUG:teuthology.orchestra.run.smithi107:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - Settings.cluster: 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'clients': ['ubuntu@smithi107.front.sepia.ceph.com'], 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'head': 'ubuntu@smithi107.front.sepia.ceph.com', 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'is_teuthology': True, 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'iterations': 1, 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'mons': ['ubuntu@smithi107.front.sepia.ceph.com'], 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'osds': ['ubuntu@smithi107.front.sepia.ceph.com'], 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'osds_per_node': 3, 2024-09-21T21:21:18.545 INFO:teuthology.orchestra.run.smithi107.stderr: 'pid_dir': '/var/run/ceph', 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'rebuild_every_test': False, 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'rgws': {'ubuntu@smithi107.front.sepia.ceph.com': None}, 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'use_existing': True, 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr: 'user': 'ubuntu'} 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2024-09-21T21:21:18.546 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2024-09-21T21:21:18.549 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-09-21T21:21:18.551 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-21T21:21:18.552 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-09-21T21:21:18.554 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:21:18.556 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-09-21T21:21:18.559 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:21:18.559 INFO:teuthology.orchestra.run.smithi107.stderr:21:21:18 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:22:18.600 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:22:18.645 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:22:18.685 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2024-09-21T21:22:18.699 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi107.front.sepia.ceph.com ; done 2024-09-21T21:22:18.703 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - INFO - cbt - Creating fio images... 2024-09-21T21:22:18.704 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-09-21T21:22:18.706 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-21T21:22:18.708 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-09-21T21:22:18.711 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-21T21:22:18.711 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-21T21:22:18.711 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-09-21T21:22:19.025 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-09-21T21:22:19.624 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-09-21T21:22:20.625 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-09-21T21:22:21.635 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-09-21T21:22:22.633 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:22 - INFO - cbt - Checking Health after pool creation. 2024-09-21T21:22:27.636 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:22:27.964 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:27 - INFO - cbt - Final Pool Health Check. 2024-09-21T21:22:32.970 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-21T21:22:33.294 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-09-21T21:22:33.296 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi107.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-09-21T21:22:34.806 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi107.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-09-21T21:22:34.920 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-21T21:22:34.945 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-21T21:22:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - INFO - cbt - Attempting to prefill fio images... 2024-09-21T21:22:34.977 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-21T21:22:34.979 INFO:teuthology.orchestra.run.smithi107.stderr:21:22:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-21T21:24:26.114 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-21T21:24:31.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~5s 2024-09-21T21:24:36.920 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2024-09-21T21:24:42.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~16s 2024-09-21T21:24:47.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2024-09-21T21:24:53.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~27s 2024-09-21T21:24:53.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-21T21:24:58.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~32s 2024-09-21T21:24:58.332 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-21T21:25:03.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~38s 2024-09-21T21:25:03.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2024-09-21T21:25:08.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~43s 2024-09-21T21:25:08.937 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~16s 2024-09-21T21:25:14.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~48s 2024-09-21T21:25:14.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~21s 2024-09-21T21:25:19.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~54s 2024-09-21T21:25:19.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~27s 2024-09-21T21:25:20.978 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:20.976+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:24.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~59s 2024-09-21T21:25:24.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~32s 2024-09-21T21:25:25.978 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:25.976+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~64s 2024-09-21T21:25:30.148 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~37s 2024-09-21T21:25:30.978 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:30.977+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:35.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~69s 2024-09-21T21:25:35.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~42s 2024-09-21T21:25:35.979 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:35.977+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:40.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~75s 2024-09-21T21:25:40.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~48s 2024-09-21T21:25:40.980 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:40.978+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:45.980 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:45.979+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:46.056 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~80s 2024-09-21T21:25:46.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~53s 2024-09-21T21:25:50.981 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:50.979+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:51.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~85s 2024-09-21T21:25:51.359 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~58s 2024-09-21T21:25:55.981 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:25:55.980+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:25:56.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~91s 2024-09-21T21:25:56.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~64s 2024-09-21T21:26:00.982 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:00.980+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:01.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~96s 2024-09-21T21:26:01.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~69s 2024-09-21T21:26:05.982 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:05.980+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:07.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~101s 2024-09-21T21:26:07.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~74s 2024-09-21T21:26:10.982 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:10.981+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:12.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~107s 2024-09-21T21:26:12.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~80s 2024-09-21T21:26:15.983 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:15.981+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:17.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~112s 2024-09-21T21:26:17.872 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~85s 2024-09-21T21:26:20.984 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:20.982+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:23.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~117s 2024-09-21T21:26:23.174 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~90s 2024-09-21T21:26:25.984 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:25.983+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:28.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~122s 2024-09-21T21:26:28.477 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~95s 2024-09-21T21:26:30.985 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:30.983+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:33.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~128s 2024-09-21T21:26:33.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~101s 2024-09-21T21:26:35.985 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:35.984+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:39.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~133s 2024-09-21T21:26:39.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~106s 2024-09-21T21:26:40.986 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:40.984+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:44.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2024-09-21T21:26:44.384 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~111s 2024-09-21T21:26:45.986 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:45.985+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:49.686 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~144s 2024-09-21T21:26:49.687 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~117s 2024-09-21T21:26:50.989 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:50.985+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:26:54.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~149s 2024-09-21T21:26:54.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~122s 2024-09-21T21:26:55.987 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:26:55.986+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:00.292 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2024-09-21T21:27:00.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2024-09-21T21:27:00.988 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:00.986+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:05.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2024-09-21T21:27:05.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~133s 2024-09-21T21:27:05.988 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:05.987+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:10.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2024-09-21T21:27:10.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2024-09-21T21:27:10.989 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:10.987+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:15.989 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:15.988+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:16.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~170s 2024-09-21T21:27:16.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2024-09-21T21:27:20.990 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:20.988+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:21.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~175s 2024-09-21T21:27:21.504 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~148s 2024-09-21T21:27:25.991 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:25.989+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:26.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~181s 2024-09-21T21:27:26.807 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2024-09-21T21:27:30.991 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:30.989+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:32.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~186s 2024-09-21T21:27:32.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~159s 2024-09-21T21:27:35.992 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:35.990+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:37.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~191s 2024-09-21T21:27:37.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~164s 2024-09-21T21:27:40.992 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:40.991+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:42.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~197s 2024-09-21T21:27:42.715 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~170s 2024-09-21T21:27:45.993 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:45.991+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:48.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~202s 2024-09-21T21:27:48.017 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~175s 2024-09-21T21:27:50.993 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:50.992+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:53.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~207s 2024-09-21T21:27:53.319 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~180s 2024-09-21T21:27:55.994 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:27:55.993+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:27:58.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~213s 2024-09-21T21:27:58.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~186s 2024-09-21T21:28:00.995 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:00.993+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:03.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~218s 2024-09-21T21:28:03.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~191s 2024-09-21T21:28:05.995 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:05.993+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:09.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~223s 2024-09-21T21:28:09.226 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~196s 2024-09-21T21:28:10.996 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:10.994+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:14.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~229s 2024-09-21T21:28:14.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~201s 2024-09-21T21:28:15.996 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:15.994+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:19.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~234s 2024-09-21T21:28:19.833 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~207s 2024-09-21T21:28:20.997 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:20.995+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:25.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~239s 2024-09-21T21:28:25.135 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~212s 2024-09-21T21:28:25.998 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:25.996+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:30.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~244s 2024-09-21T21:28:30.437 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~217s 2024-09-21T21:28:30.998 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:30.997+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:35.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~250s 2024-09-21T21:28:35.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~223s 2024-09-21T21:28:35.999 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:35.997+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:40.999 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:40.998+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:41.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~255s 2024-09-21T21:28:41.043 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~228s 2024-09-21T21:28:46.000 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:45.999+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:46.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~260s 2024-09-21T21:28:46.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~233s 2024-09-21T21:28:51.001 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:50.999+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:51.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~266s 2024-09-21T21:28:51.649 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~239s 2024-09-21T21:28:56.001 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:28:56.000+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:28:56.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~271s 2024-09-21T21:28:56.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~244s 2024-09-21T21:29:01.002 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:01.000+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:02.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~276s 2024-09-21T21:29:02.254 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~249s 2024-09-21T21:29:06.004 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:06.002+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:07.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~282s 2024-09-21T21:29:07.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~255s 2024-09-21T21:29:11.005 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:11.003+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:12.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~287s 2024-09-21T21:29:12.860 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~260s 2024-09-21T21:29:16.006 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:16.004+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:18.163 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2024-09-21T21:29:18.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~265s 2024-09-21T21:29:21.006 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:21.004+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:23.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2024-09-21T21:29:23.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2024-09-21T21:29:26.007 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:26.005+0000 7f3940df9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.0 v2:172.21.15.107:6802/3908228775 for 29sec e20 v20) 2024-09-21T21:29:28.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2024-09-21T21:29:28.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~276s 2024-09-21T21:29:28.769 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-09-21T21:29:29.071 INFO:tasks.ceph.osd.1:Sent signal 15 2024-09-21T21:29:29.072 INFO:tasks.ceph.mon.a:Sent signal 15 2024-09-21T21:29:29.072 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-09-21T21:29:29.073 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:29.071+0000 7f3943dff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 29009) UID: 0 2024-09-21T21:29:29.073 INFO:tasks.ceph.mon.a.smithi107.stderr:2024-09-21T21:29:29.071+0000 7f3943dff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T21:29:29.073 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:29:29.071+0000 7fcd8f9f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 29012) UID: 0 2024-09-21T21:29:29.073 INFO:tasks.ceph.mgr.x.smithi107.stderr:2024-09-21T21:29:29.071+0000 7fcd8f9f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-22T05:10:59.371 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-22T05:10:59.402 DEBUG:teuthology.task.console_log:Killing console logger for smithi107 2024-09-22T05:10:59.403 DEBUG:teuthology.exit:Finished running handlers