2024-09-22T03:38:21.078 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:38:21.097 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:38:21.171 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914376 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/fio_4M_rand_read} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914376' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi083.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDouoPibb/w941bFOT2pFmswA0CLROENd3gZt+9bAAXjEijlKBgloJcGS8bpKBXmyd/BED1UtYXw/14ynAzBdjU= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:38:21.172 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-22T03:38:21.172 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-22T03:38:21.172 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:38:21.174 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:38:21.193 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-22T03:38:21.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:38:21.193 INFO:teuthology.packaging:ref: None 2024-09-22T03:38:21.193 INFO:teuthology.packaging:tag: None 2024-09-22T03:38:21.194 INFO:teuthology.packaging:branch: main 2024-09-22T03:38:21.194 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:38:21.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-22T03:38:21.409 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-22T03:38:21.411 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:38:21.418 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:38:21.418 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:38:21.459 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:38:21.474 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:38:21.481 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:38:21.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi083.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914376', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:33:24.085949', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDouoPibb/w941bFOT2pFmswA0CLROENd3gZt+9bAAXjEijlKBgloJcGS8bpKBXmyd/BED1UtYXw/14ynAzBdjU='} 2024-09-22T03:38:21.506 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:38:21.512 INFO:teuthology.task.internal:roles: ubuntu@smithi083.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:38:21.512 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:38:21.546 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe086938160>, signals=[15]) 2024-09-22T03:38:21.546 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:38:21.551 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:38:21.552 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi083.front.sepia.ceph.com 2024-09-22T03:38:21.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:38:21.628 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:38:21.634 DEBUG:teuthology.orchestra.run.smithi083:> uname -m 2024-09-22T03:38:21.712 INFO:teuthology.orchestra.run.smithi083.stdout:x86_64 2024-09-22T03:38:21.713 DEBUG:teuthology.orchestra.run.smithi083:> cat /etc/os-release 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:NAME="CentOS Stream" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION="9" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:ID="centos" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:VERSION_ID="9" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:ANSI_COLOR="0;31" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:38:21.770 INFO:teuthology.orchestra.run.smithi083.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:38:21.771 INFO:teuthology.orchestra.run.smithi083.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:38:21.771 INFO:teuthology.orchestra.run.smithi083.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:38:21.771 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:38:21.771 INFO:teuthology.orchestra.run.smithi083.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:38:21.771 INFO:teuthology.lock.ops:Updating smithi083.front.sepia.ceph.com on lock server 2024-09-22T03:38:21.795 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:38:21.803 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:38:21.809 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:38:21.810 DEBUG:teuthology.orchestra.run.smithi083:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:38:21.826 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:38:21.832 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:38:21.832 DEBUG:teuthology.orchestra.run.smithi083:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:38:21.896 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:38:21.929 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:38:21.943 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:38:21.943 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:38:21.943 DEBUG:teuthology.orchestra.run.smithi083:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:38:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:38:21.960 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-22T03:38:22.016 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:38:22.016 INFO:teuthology.task.kernel:Running kernel on smithi083: 5.14.0-503.el9.x86_64 2024-09-22T03:38:22.016 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum install -y kernel 2024-09-22T03:38:25.743 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.3 MB 00:01 2024-09-22T03:38:29.569 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 7.4 MB/s | 20 MB 00:02 2024-09-22T03:38:35.028 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-22T03:38:36.437 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 92 kB/s | 19 kB 00:00 2024-09-22T03:38:37.983 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-22T03:38:43.372 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-22T03:38:45.311 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:38:45.312 INFO:teuthology.orchestra.run.smithi083.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:38:45.407 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:38:45.422 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:38:45.422 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:38:45.423 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:38:45.424 INFO:teuthology.orchestra.run.smithi083.stdout:Install 4 Packages 2024-09-22T03:38:45.424 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:38:45.424 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 79 M 2024-09-22T03:38:45.424 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 126 M 2024-09-22T03:38:45.424 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:38:45.757 INFO:teuthology.orchestra.run.smithi083.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 452 kB/s | 36 kB 00:00 2024-09-22T03:38:46.290 INFO:teuthology.orchestra.run.smithi083.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-09-22T03:38:46.982 INFO:teuthology.orchestra.run.smithi083.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 26 MB/s | 35 MB 00:01 2024-09-22T03:38:47.908 INFO:teuthology.orchestra.run.smithi083.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-22T03:38:47.908 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:47.908 INFO:teuthology.orchestra.run.smithi083.stdout:Total 32 MB/s | 79 MB 00:02 2024-09-22T03:38:48.587 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:38:48.944 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:38:48.944 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:38:49.882 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:38:49.883 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:38:52.324 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:38:53.020 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:38:53.102 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:38:55.837 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:38:55.971 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:39:00.794 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:39:00.905 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:05.767 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:44.455 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:44.496 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:44.791 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:44.791 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:39:44.791 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:39:44.791 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:39:45.592 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:39:45.592 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:39:45.592 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:39:45.593 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:39:45.939 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:39:46.031 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:39:46.031 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:39:46.031 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:39:46.031 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:39:46.031 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:39:46.032 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:39:46.032 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi083.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:39:46.032 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:39:46.089 INFO:teuthology.orchestra.run.smithi083.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:39:46.090 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:39:46.090 DEBUG:teuthology.orchestra.run.smithi083:> sudo rpm -qi grub2-tools 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Name : grub2-tools 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Epoch : 1 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Version : 2.06 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Release : 82.el9 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Architecture: x86_64 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Group : Unspecified 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:Size : 8274670 2024-09-22T03:39:46.129 INFO:teuthology.orchestra.run.smithi083.stdout:License : GPLv3+ 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Packager : builder@centos.org 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Vendor : CentOS 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Summary : Support tools for GRUB. 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:Description : 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout:hardware devices. 2024-09-22T03:39:46.130 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:39:46.131 INFO:teuthology.orchestra.run.smithi083.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:39:46.131 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:39:46.131 INFO:teuthology.task.kernel:Updating grub on smithi083 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:39:46.131 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:39:46.744 INFO:teuthology.orchestra.run.smithi083.stderr:Generating grub configuration file ... 2024-09-22T03:39:48.003 INFO:teuthology.orchestra.run.smithi083.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:39:48.049 INFO:teuthology.orchestra.run.smithi083.stderr:done 2024-09-22T03:39:48.051 DEBUG:teuthology.orchestra.run.smithi083:> mktemp 2024-09-22T03:39:48.066 INFO:teuthology.orchestra.run.smithi083.stdout:/tmp/tmp.ABJ4BN3RxS 2024-09-22T03:39:48.067 DEBUG:teuthology.orchestra.run.smithi083:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ABJ4BN3RxS 2024-09-22T03:39:48.132 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0666 /tmp/tmp.ABJ4BN3RxS 2024-09-22T03:39:48.281 DEBUG:teuthology.orchestra.remote:smithi083:/tmp/tmp.ABJ4BN3RxS is 6KB 2024-09-22T03:39:48.331 DEBUG:teuthology.orchestra.run.smithi083:> rm -fr /tmp/tmp.ABJ4BN3RxS 2024-09-22T03:39:48.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:39:48.415 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:39:48.415 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:39:48.415 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:39:48.415 INFO:teuthology.orchestra.run.smithi083.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:39:48.417 DEBUG:teuthology.orchestra.run.smithi083:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:39:48.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo shutdown -r now 2024-09-22T03:40:18.598 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:40:18.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:40:18.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:40:36.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-22T03:40:45.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:40:45.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:40:49.042 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-22T03:41:01.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:41:01.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:41:19.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.83 2024-09-22T03:41:34.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:41:34.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:41:34.804 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-22T03:41:35.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:41:35.200 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:41:35.200 DEBUG:teuthology.orchestra.run.smithi083:> uname -r 2024-09-22T03:41:35.215 INFO:teuthology.orchestra.run.smithi083.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:41:35.215 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:41:35.215 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:41:35.215 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:41:36.217 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:41:36.217 DEBUG:teuthology.orchestra.run.smithi083:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:41:36.318 INFO:teuthology.orchestra.run.smithi083.stdout:ttyS1 2024-09-22T03:41:36.339 DEBUG:teuthology.parallel:result is None 2024-09-22T03:41:36.340 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:41:36.348 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:41:36.348 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:41:36.390 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:41:36.396 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:41:36.470 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:41:36.471 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:41:36.515 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:41:36.521 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:41:36.521 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:41:36.610 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:41:36.629 INFO:teuthology.orchestra.run.smithi083.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:41:36.631 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:41:36.638 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:41:36.638 DEBUG:teuthology.orchestra.run.smithi083:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:41:36.691 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:41:36.701 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:41:36.701 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:41:36.728 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:41:36.827 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:41:36.912 DEBUG:teuthology.orchestra.run.smithi083:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:41:36.988 DEBUG:teuthology.orchestra.run.smithi083:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:41:37.060 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:41:37.060 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:41:37.089 DEBUG:teuthology.orchestra.run.smithi083:> sudo service rsyslog restart 2024-09-22T03:41:37.175 INFO:teuthology.orchestra.run.smithi083.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:41:37.512 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:41:37.519 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:41:37.519 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:41:37.531 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:41:37.538 DEBUG:teuthology.orchestra.run.smithi083:> sudo service auditd rotate 2024-09-22T03:41:37.711 INFO:teuthology.orchestra.run.smithi083.stdout:Rotating logs: 2024-09-22T03:41:37.713 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:41:37.713 DEBUG:teuthology.orchestra.run.smithi083:> /usr/sbin/getenforce 2024-09-22T03:41:37.748 INFO:teuthology.orchestra.run.smithi083.stdout:Permissive 2024-09-22T03:41:37.748 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi083.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:41:37.748 DEBUG:teuthology.orchestra.run.smithi083:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T03:41:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:41:37.811 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:41:37.811 DEBUG:teuthology.orchestra.run.smithi083:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:41:37.898 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:41:37.912 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:41:37.922 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:41:37.923 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi083.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:45:20.709 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi083.front.sepia.ceph.com')] 2024-09-22T03:45:20.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:45:20.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi083.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:45:20.792 DEBUG:teuthology.orchestra.run.smithi083:> true 2024-09-22T03:45:20.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi083.front.sepia.ceph.com' 2024-09-22T03:45:20.876 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:45:20.888 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:45:20.889 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:45:20.889 DEBUG:teuthology.orchestra.run.smithi083:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:45:20.948 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:45:20.963 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:45:20.989 INFO:teuthology.orchestra.run.smithi083.stderr:sudo: ntpd: command not found 2024-09-22T03:45:21.000 INFO:teuthology.orchestra.run.smithi083.stdout:506 Cannot talk to daemon 2024-09-22T03:45:21.015 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:45:21.029 INFO:teuthology.orchestra.run.smithi083.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:45:21.089 INFO:teuthology.orchestra.run.smithi083.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:45:21.092 INFO:teuthology.orchestra.run.smithi083.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:45:21.093 INFO:teuthology.orchestra.run.smithi083.stdout:=============================================================================== 2024-09-22T03:45:21.093 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:45:21.093 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:45:21.093 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:45:21.093 INFO:teuthology.orchestra.run.smithi083.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:45:21.094 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:45:21.101 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:45:21.101 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-22T03:45:21.102 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-22T03:45:21.102 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:45:21.107 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:45:21.107 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:45:21.107 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-22T03:45:21.108 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:45:21.253 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-22T03:45:21.254 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-22T03:45:21.365 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:45:21.365 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:45:21.366 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:45:21.392 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:45:21.392 DEBUG:teuthology.orchestra.run.smithi083:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:45:21.458 DEBUG:teuthology.orchestra.run.smithi083:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:45:21.533 DEBUG:teuthology.orchestra.run.smithi083:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:45:21.597 INFO:teuthology.orchestra.run.smithi083.stdout:check_obsoletes = 1 2024-09-22T03:45:21.599 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum clean all 2024-09-22T03:45:21.827 INFO:teuthology.orchestra.run.smithi083.stdout:45 files removed 2024-09-22T03:45:21.855 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-radosgw 2024-09-22T03:45:22.399 INFO:teuthology.orchestra.run.smithi083.stdout:ceph packages for x86_64 341 kB/s | 88 kB 00:00 2024-09-22T03:45:22.675 INFO:teuthology.orchestra.run.smithi083.stdout:ceph noarch packages 105 kB/s | 24 kB 00:00 2024-09-22T03:45:22.943 INFO:teuthology.orchestra.run.smithi083.stdout:ceph source packages 9.0 kB/s | 2.1 kB 00:00 2024-09-22T03:45:23.459 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-22T03:45:25.386 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-09-22T03:45:30.853 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-22T03:45:32.437 INFO:teuthology.orchestra.run.smithi083.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 19 kB 00:00 2024-09-22T03:45:33.900 INFO:teuthology.orchestra.run.smithi083.stdout:Extra Packages for Enterprise Linux 17 MB/s | 23 MB 00:01 2024-09-22T03:45:39.323 INFO:teuthology.orchestra.run.smithi083.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-22T03:45:41.575 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:45:41.590 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrading: 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-22T03:45:41.591 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:45:41.592 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout:Installing weak dependencies: 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:45:41.593 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:45:41.594 INFO:teuthology.orchestra.run.smithi083.stdout:Install 31 Packages 2024-09-22T03:45:41.594 INFO:teuthology.orchestra.run.smithi083.stdout:Upgrade 2 Packages 2024-09-22T03:45:41.594 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:45:41.594 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 60 M 2024-09-22T03:45:41.594 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:45:42.708 INFO:teuthology.orchestra.run.smithi083.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 9.0 MB/s | 6.1 MB 00:00 2024-09-22T03:45:42.749 INFO:teuthology.orchestra.run.smithi083.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 608 kB/s | 25 kB 00:00 2024-09-22T03:45:42.833 INFO:teuthology.orchestra.run.smithi083.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 9.5 MB/s | 808 kB 00:00 2024-09-22T03:45:42.975 INFO:teuthology.orchestra.run.smithi083.stdout:(4/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 11 MB/s | 10 MB 00:00 2024-09-22T03:45:43.009 INFO:teuthology.orchestra.run.smithi083.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 3.1 MB/s | 558 kB 00:00 2024-09-22T03:45:43.067 INFO:teuthology.orchestra.run.smithi083.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 772 kB/s | 45 kB 00:00 2024-09-22T03:45:43.142 INFO:teuthology.orchestra.run.smithi083.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 1.9 MB/s | 148 kB 00:00 2024-09-22T03:45:43.235 INFO:teuthology.orchestra.run.smithi083.stdout:(8/33): python3-cephfs-19.3.0-5092.g62da54b4.el 1.7 MB/s | 159 kB 00:00 2024-09-22T03:45:43.302 INFO:teuthology.orchestra.run.smithi083.stdout:(9/33): python3-rados-19.3.0-5092.g62da54b4.el9 4.6 MB/s | 314 kB 00:00 2024-09-22T03:45:43.385 INFO:teuthology.orchestra.run.smithi083.stdout:(10/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_ 13 MB/s | 5.4 MB 00:00 2024-09-22T03:45:43.419 INFO:teuthology.orchestra.run.smithi083.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 2.5 MB/s | 302 kB 00:00 2024-09-22T03:45:43.444 INFO:teuthology.orchestra.run.smithi083.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-22T03:45:43.636 INFO:teuthology.orchestra.run.smithi083.stdout:(13/33): ceph-common-19.3.0-5092.g62da54b4.el9. 12 MB/s | 20 MB 00:01 2024-09-22T03:45:44.228 INFO:teuthology.orchestra.run.smithi083.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 132 kB/s | 107 kB 00:00 2024-09-22T03:45:44.287 INFO:teuthology.orchestra.run.smithi083.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 202 kB/s | 171 kB 00:00 2024-09-22T03:45:44.420 INFO:teuthology.orchestra.run.smithi083.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 204 kB/s | 160 kB 00:00 2024-09-22T03:45:44.804 INFO:teuthology.orchestra.run.smithi083.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 79 kB/s | 45 kB 00:00 2024-09-22T03:45:45.179 INFO:teuthology.orchestra.run.smithi083.stdout:(18/33): librdkafka-1.6.1-102.el9.x86_64.rpm 743 kB/s | 662 kB 00:00 2024-09-22T03:45:45.204 INFO:teuthology.orchestra.run.smithi083.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 373 kB/s | 292 kB 00:00 2024-09-22T03:45:45.380 INFO:teuthology.orchestra.run.smithi083.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 73 kB/s | 42 kB 00:00 2024-09-22T03:45:45.788 INFO:teuthology.orchestra.run.smithi083.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 61 kB/s | 36 kB 00:00 2024-09-22T03:45:45.880 INFO:teuthology.orchestra.run.smithi083.stdout:(22/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.3 MB/s | 308 kB 00:00 2024-09-22T03:45:45.923 INFO:teuthology.orchestra.run.smithi083.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 297 kB/s | 221 kB 00:00 2024-09-22T03:45:45.982 INFO:teuthology.orchestra.run.smithi083.stdout:(24/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 79 kB/s | 47 kB 00:00 2024-09-22T03:45:46.015 INFO:teuthology.orchestra.run.smithi083.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2024-09-22T03:45:46.048 INFO:teuthology.orchestra.run.smithi083.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 758 kB/s | 49 kB 00:00 2024-09-22T03:45:46.082 INFO:teuthology.orchestra.run.smithi083.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-22T03:45:46.124 INFO:teuthology.orchestra.run.smithi083.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-22T03:45:46.166 INFO:teuthology.orchestra.run.smithi083.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-22T03:45:46.250 INFO:teuthology.orchestra.run.smithi083.stdout:(30/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-22T03:45:46.325 INFO:teuthology.orchestra.run.smithi083.stdout:(31/33): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-09-22T03:45:46.508 INFO:teuthology.orchestra.run.smithi083.stdout:(32/33): librados2-19.3.0-5092.g62da54b4.el9.x8 10 MB/s | 3.5 MB 00:00 2024-09-22T03:45:46.583 INFO:teuthology.orchestra.run.smithi083.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 11 MB/s | 3.8 MB 00:00 2024-09-22T03:45:46.589 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:45:46.590 INFO:teuthology.orchestra.run.smithi083.stdout:Total 12 MB/s | 60 MB 00:04 2024-09-22T03:45:46.709 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:45:46.745 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:45:46.746 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:45:47.437 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:45:47.437 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:45:48.924 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:45:49.442 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-22T03:45:49.511 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:45:49.586 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:45:49.683 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:45:49.712 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:45:49.767 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:45:49.821 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-22T03:45:49.850 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-22T03:45:49.876 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-22T03:45:49.898 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:45:49.917 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:45:50.109 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:45:50.140 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:45:50.175 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-22T03:45:50.227 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-22T03:45:50.256 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:45:50.335 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-22T03:45:50.361 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:45:50.450 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:45:50.486 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-22T03:45:50.598 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-22T03:45:50.628 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-22T03:45:50.686 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-22T03:45:50.718 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-22T03:45:50.747 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-22T03:45:50.785 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-22T03:45:51.165 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-22T03:45:51.218 INFO:teuthology.orchestra.run.smithi083.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:45:51.310 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:45:51.340 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-22T03:45:51.376 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-22T03:45:51.973 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-22T03:45:52.004 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:45:52.076 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:45:52.158 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-22T03:45:52.199 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-22T03:45:54.614 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:45:54.653 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:45:55.379 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:45:55.424 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:45:55.497 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:45:55.497 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:45:55.497 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:45:55.497 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:45:55.541 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /sys 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /proc 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /mnt 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /var/tmp 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /home 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /root 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout:skipping the directory /tmp 2024-09-22T03:46:05.891 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:06.996 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:46:07.021 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:07.023 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:46:07.123 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:46:07.123 INFO:teuthology.orchestra.run.smithi083.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:46:08.463 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:46:08.464 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-22T03:46:08.465 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-22T03:46:08.466 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout:Upgraded: 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:46:09.459 INFO:teuthology.orchestra.run.smithi083.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:46:09.460 INFO:teuthology.orchestra.run.smithi083.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:46:09.461 INFO:teuthology.orchestra.run.smithi083.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:46:09.462 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:09.462 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:46:09.817 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-test 2024-09-22T03:46:10.394 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:31 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:46:10.525 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:46:10.525 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:46:10.526 INFO:teuthology.orchestra.run.smithi083.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout:Install 9 Packages 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 83 M 2024-09-22T03:46:10.527 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 775 M 2024-09-22T03:46:10.528 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:46:11.337 INFO:teuthology.orchestra.run.smithi083.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 663 kB/s | 94 kB 00:00 2024-09-22T03:46:11.404 INFO:teuthology.orchestra.run.smithi083.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 837 kB/s | 175 kB 00:00 2024-09-22T03:46:11.579 INFO:teuthology.orchestra.run.smithi083.stdout:(3/9): libubsan-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 183 kB 00:00 2024-09-22T03:46:11.612 INFO:teuthology.orchestra.run.smithi083.stdout:(4/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 1.8 MB/s | 513 kB 00:00 2024-09-22T03:46:11.671 INFO:teuthology.orchestra.run.smithi083.stdout:(5/9): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-09-22T03:46:11.730 INFO:teuthology.orchestra.run.smithi083.stdout:(6/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-22T03:46:11.755 INFO:teuthology.orchestra.run.smithi083.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 761 kB/s | 64 kB 00:00 2024-09-22T03:46:11.805 INFO:teuthology.orchestra.run.smithi083.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.6 MB/s | 122 kB 00:00 2024-09-22T03:46:13.564 INFO:teuthology.orchestra.run.smithi083.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 34 MB/s | 81 MB 00:02 2024-09-22T03:46:13.567 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:46:13.568 INFO:teuthology.orchestra.run.smithi083.stdout:Total 27 MB/s | 83 MB 00:03 2024-09-22T03:46:13.653 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:46:13.676 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:46:13.676 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:46:14.264 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:46:14.265 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:46:15.115 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:46:15.193 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-22T03:46:15.242 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-22T03:46:15.285 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-22T03:46:15.325 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-22T03:46:15.360 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:46:15.384 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:46:15.454 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:46:15.501 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-22T03:46:15.528 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:46:26.491 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:46:26.568 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:46:32.788 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-22T03:46:32.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-22T03:46:32.789 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-22T03:46:33.213 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:46:33.214 INFO:teuthology.orchestra.run.smithi083.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:46:33.214 INFO:teuthology.orchestra.run.smithi083.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:46:33.214 INFO:teuthology.orchestra.run.smithi083.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:46:33.214 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:33.214 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:46:33.578 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph 2024-09-22T03:46:34.158 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:00:55 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:46:34.299 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:46:34.301 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:34.301 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:46:34.301 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:46:34.302 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:46:34.303 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:46:34.304 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout:Install 37 Packages 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:34.305 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 84 M 2024-09-22T03:46:34.306 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 896 M 2024-09-22T03:46:34.306 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:46:35.207 INFO:teuthology.orchestra.run.smithi083.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 18 kB/s | 6.5 kB 00:00 2024-09-22T03:46:35.790 INFO:teuthology.orchestra.run.smithi083.stdout:(2/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 2.3 MB/s | 2.2 MB 00:00 2024-09-22T03:46:36.599 INFO:teuthology.orchestra.run.smithi083.stdout:(3/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 3.7 MB/s | 5.1 MB 00:01 2024-09-22T03:46:36.649 INFO:teuthology.orchestra.run.smithi083.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 5.7 MB/s | 292 kB 00:00 2024-09-22T03:46:36.691 INFO:teuthology.orchestra.run.smithi083.stdout:(5/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 887 kB/s | 1.6 MB 00:01 2024-09-22T03:46:36.816 INFO:teuthology.orchestra.run.smithi083.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-22T03:46:36.858 INFO:teuthology.orchestra.run.smithi083.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-22T03:46:36.900 INFO:teuthology.orchestra.run.smithi083.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-22T03:46:36.942 INFO:teuthology.orchestra.run.smithi083.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-22T03:46:36.984 INFO:teuthology.orchestra.run.smithi083.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 4.2 MB/s | 1.2 MB 00:00 2024-09-22T03:46:37.009 INFO:teuthology.orchestra.run.smithi083.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-22T03:46:37.143 INFO:teuthology.orchestra.run.smithi083.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-22T03:46:37.168 INFO:teuthology.orchestra.run.smithi083.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-22T03:46:37.185 INFO:teuthology.orchestra.run.smithi083.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 829 kB/s | 35 kB 00:00 2024-09-22T03:46:37.219 INFO:teuthology.orchestra.run.smithi083.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-22T03:46:37.236 INFO:teuthology.orchestra.run.smithi083.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-22T03:46:37.261 INFO:teuthology.orchestra.run.smithi083.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 992 kB/s | 42 kB 00:00 2024-09-22T03:46:37.320 INFO:teuthology.orchestra.run.smithi083.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 2.5 MB/s | 151 kB 00:00 2024-09-22T03:46:37.337 INFO:teuthology.orchestra.run.smithi083.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.6 MB/s | 43 kB 00:00 2024-09-22T03:46:37.362 INFO:teuthology.orchestra.run.smithi083.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-09-22T03:46:37.396 INFO:teuthology.orchestra.run.smithi083.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-22T03:46:37.413 INFO:teuthology.orchestra.run.smithi083.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 649 kB/s | 11 kB 00:00 2024-09-22T03:46:37.438 INFO:teuthology.orchestra.run.smithi083.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-22T03:46:37.455 INFO:teuthology.orchestra.run.smithi083.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 1.3 MB/s | 23 kB 00:00 2024-09-22T03:46:37.472 INFO:teuthology.orchestra.run.smithi083.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-09-22T03:46:37.498 INFO:teuthology.orchestra.run.smithi083.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 1.1 MB/s | 20 kB 00:00 2024-09-22T03:46:37.514 INFO:teuthology.orchestra.run.smithi083.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 2.7 MB/s | 46 kB 00:00 2024-09-22T03:46:37.548 INFO:teuthology.orchestra.run.smithi083.stdout:(28/37): python3-more-itertools-8.12.0-2.el9.no 2.3 MB/s | 79 kB 00:00 2024-09-22T03:46:37.590 INFO:teuthology.orchestra.run.smithi083.stdout:(29/37): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-09-22T03:46:37.607 INFO:teuthology.orchestra.run.smithi083.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 998 kB/s | 16 kB 00:00 2024-09-22T03:46:37.632 INFO:teuthology.orchestra.run.smithi083.stdout:(31/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.5 MB/s | 90 kB 00:00 2024-09-22T03:46:37.658 INFO:teuthology.orchestra.run.smithi083.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-22T03:46:37.691 INFO:teuthology.orchestra.run.smithi083.stdout:(33/37): python3-webob-1.8.8-2.el9.noarch.rpm 6.8 MB/s | 230 kB 00:00 2024-09-22T03:46:37.725 INFO:teuthology.orchestra.run.smithi083.stdout:(34/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 12 MB/s | 427 kB 00:00 2024-09-22T03:46:37.750 INFO:teuthology.orchestra.run.smithi083.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-22T03:46:37.800 INFO:teuthology.orchestra.run.smithi083.stdout:(36/37): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-09-22T03:46:38.609 INFO:teuthology.orchestra.run.smithi083.stdout:(37/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 25 MB/s | 70 MB 00:02 2024-09-22T03:46:38.613 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:46:38.613 INFO:teuthology.orchestra.run.smithi083.stdout:Total 20 MB/s | 84 MB 00:04 2024-09-22T03:46:38.768 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:46:38.809 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:46:38.809 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:46:39.623 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:46:39.623 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:46:41.246 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:46:41.295 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-22T03:46:41.335 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-22T03:46:41.366 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-22T03:46:41.439 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-22T03:46:41.482 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-22T03:46:41.515 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-22T03:46:41.607 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-22T03:46:41.694 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-22T03:46:41.734 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-22T03:46:41.769 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-22T03:46:41.799 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-22T03:46:41.825 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-22T03:46:41.850 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-22T03:46:41.891 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-22T03:46:41.926 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-22T03:46:41.960 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-22T03:46:42.008 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-22T03:46:42.041 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-22T03:46:42.114 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-22T03:46:42.262 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-22T03:46:42.342 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-22T03:46:42.394 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-22T03:46:42.432 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-22T03:46:42.489 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-22T03:46:42.556 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-22T03:46:42.746 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-22T03:46:42.805 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-22T03:46:42.892 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-22T03:46:42.990 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-22T03:46:43.040 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-22T03:46:43.174 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-22T03:46:43.582 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-22T03:46:43.607 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:46:43.632 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:46:43.632 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:46:43.632 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:46:43.633 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:46:43.633 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:46:43.633 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:54.149 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:46:54.175 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:46:54.175 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:46:54.176 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:46:54.176 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:46:54.176 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:46:54.176 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:54.733 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:46:54.756 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:46:54.756 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:46:54.757 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:46:54.757 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:46:54.757 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:46:54.757 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:55.000 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:46:55.024 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:46:55.097 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:47:01.804 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-22T03:47:01.805 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-22T03:47:01.806 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-22T03:47:01.807 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-22T03:47:02.898 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-22T03:47:02.898 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:02.898 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:02.898 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:02.898 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:47:02.899 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:47:02.900 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:02.901 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:03.214 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-base 2024-09-22T03:47:03.790 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:24 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:03.879 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:03.922 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:03.923 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:03.923 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:04.001 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install cephadm 2024-09-22T03:47:04.572 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:25 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:04.702 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:04.702 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:04.702 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:04.702 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 784 k 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 791 k 2024-09-22T03:47:04.703 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:04.967 INFO:teuthology.orchestra.run.smithi083.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-22T03:47:04.967 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:04.968 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-22T03:47:04.968 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:04.973 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:04.973 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:04.979 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:04.979 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:05.059 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:05.495 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:05.570 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:05.979 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:06.300 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:06.531 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:47:07.119 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:28 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:07.253 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:07.253 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:07.253 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 158 k 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 644 k 2024-09-22T03:47:07.254 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:07.599 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 458 kB/s | 158 kB 00:00 2024-09-22T03:47:07.599 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:07.600 INFO:teuthology.orchestra.run.smithi083.stdout:Total 456 kB/s | 158 kB 00:00 2024-09-22T03:47:07.600 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:07.609 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:07.609 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:07.666 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:07.666 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:07.812 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:07.875 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:47:07.900 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:47:07.900 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:47:07.900 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:47:07.900 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:08.841 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:09.076 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr 2024-09-22T03:47:09.667 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:30 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:09.758 INFO:teuthology.orchestra.run.smithi083.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:09.799 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:09.800 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:09.801 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:09.867 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:47:10.445 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:31 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:10.576 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-22T03:47:10.577 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:10.578 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:10.579 INFO:teuthology.orchestra.run.smithi083.stdout:Install 11 Packages 2024-09-22T03:47:10.579 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:10.579 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 18 M 2024-09-22T03:47:10.579 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 156 M 2024-09-22T03:47:10.579 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:11.095 INFO:teuthology.orchestra.run.smithi083.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 159 kB/s | 34 kB 00:00 2024-09-22T03:47:11.112 INFO:teuthology.orchestra.run.smithi083.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 71 kB/s | 16 kB 00:00 2024-09-22T03:47:11.312 INFO:teuthology.orchestra.run.smithi083.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.2 MB/s | 269 kB 00:00 2024-09-22T03:47:11.387 INFO:teuthology.orchestra.run.smithi083.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 7.1 MB/s | 548 kB 00:00 2024-09-22T03:47:11.413 INFO:teuthology.orchestra.run.smithi083.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 770 kB/s | 19 kB 00:00 2024-09-22T03:47:11.555 INFO:teuthology.orchestra.run.smithi083.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-09-22T03:47:11.572 INFO:teuthology.orchestra.run.smithi083.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 8.5 MB/s | 144 kB 00:00 2024-09-22T03:47:11.597 INFO:teuthology.orchestra.run.smithi083.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-22T03:47:11.631 INFO:teuthology.orchestra.run.smithi083.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 5.6 MB/s | 188 kB 00:00 2024-09-22T03:47:12.014 INFO:teuthology.orchestra.run.smithi083.stdout:(10/11): protobuf-compiler-3.14.0-13.el9.x86_64 957 kB/s | 863 kB 00:00 2024-09-22T03:47:13.406 INFO:teuthology.orchestra.run.smithi083.stdout:(11/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 5.6 MB/s | 14 MB 00:02 2024-09-22T03:47:13.410 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:13.410 INFO:teuthology.orchestra.run.smithi083.stdout:Total 6.5 MB/s | 18 MB 00:02 2024-09-22T03:47:13.440 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:13.459 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:13.460 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:13.734 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:13.734 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:14.318 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:14.412 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:47:14.434 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:47:14.542 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:47:14.676 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:47:14.730 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:47:14.813 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:47:14.865 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:47:14.894 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:47:14.926 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-22T03:47:18.095 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-22T03:47:18.155 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:47:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:47:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-22T03:47:20.069 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:47:20.070 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:20.371 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:20.372 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:20.650 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:47:21.218 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:42 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:21.350 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:47:21.351 INFO:teuthology.orchestra.run.smithi083.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout:======================================================================================= 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout:Install 12 Packages 2024-09-22T03:47:21.352 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:21.353 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 38 M 2024-09-22T03:47:21.353 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 212 M 2024-09-22T03:47:21.353 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:22.179 INFO:teuthology.orchestra.run.smithi083.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-50 16 MB/s | 7.4 MB 00:00 2024-09-22T03:47:22.246 INFO:teuthology.orchestra.run.smithi083.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 492 kB/s | 33 kB 00:00 2024-09-22T03:47:22.271 INFO:teuthology.orchestra.run.smithi083.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 340 kB/s | 191 kB 00:00 2024-09-22T03:47:22.338 INFO:teuthology.orchestra.run.smithi083.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 273 kB/s | 18 kB 00:00 2024-09-22T03:47:22.397 INFO:teuthology.orchestra.run.smithi083.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 661 kB/s | 38 kB 00:00 2024-09-22T03:47:22.447 INFO:teuthology.orchestra.run.smithi083.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-09-22T03:47:22.489 INFO:teuthology.orchestra.run.smithi083.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 5.7 MB/s | 245 kB 00:00 2024-09-22T03:47:22.640 INFO:teuthology.orchestra.run.smithi083.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-22T03:47:22.706 INFO:teuthology.orchestra.run.smithi083.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-22T03:47:22.741 INFO:teuthology.orchestra.run.smithi083.stdout:(10/12): libgfortran-11.5.0-2.el9.x86_64.rpm 777 kB/s | 801 kB 00:01 2024-09-22T03:47:22.765 INFO:teuthology.orchestra.run.smithi083.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-09-22T03:47:23.049 INFO:teuthology.orchestra.run.smithi083.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-09-22T03:47:23.049 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:23.050 INFO:teuthology.orchestra.run.smithi083.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-22T03:47:23.407 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:23.434 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:23.434 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:23.857 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:23.857 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:24.618 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:24.690 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:47:24.730 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:47:24.829 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:47:24.865 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:47:25.048 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:47:25.080 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:47:25.157 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:47:25.818 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:47:25.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:47:27.179 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:47:28.202 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:47:28.222 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:47:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:47:30.867 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:47:30.868 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:47:31.331 INFO:teuthology.orchestra.run.smithi083.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:31.332 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:31.714 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:47:32.304 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:53 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:32.437 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-22T03:47:32.438 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:32.439 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout:Install 13 Packages 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.2 M 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 26 M 2024-09-22T03:47:32.440 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:32.691 INFO:teuthology.orchestra.run.smithi083.stdout:(1/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 287 kB/s | 49 kB 00:00 2024-09-22T03:47:32.716 INFO:teuthology.orchestra.run.smithi083.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 130 kB/s | 26 kB 00:00 2024-09-22T03:47:32.742 INFO:teuthology.orchestra.run.smithi083.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 86 kB/s | 19 kB 00:00 2024-09-22T03:47:33.000 INFO:teuthology.orchestra.run.smithi083.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 615 kB/s | 159 kB 00:00 2024-09-22T03:47:33.025 INFO:teuthology.orchestra.run.smithi083.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 665 kB/s | 222 kB 00:00 2024-09-22T03:47:33.067 INFO:teuthology.orchestra.run.smithi083.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 771 kB/s | 32 kB 00:00 2024-09-22T03:47:33.093 INFO:teuthology.orchestra.run.smithi083.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 562 kB/s | 14 kB 00:00 2024-09-22T03:47:33.118 INFO:teuthology.orchestra.run.smithi083.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 455 kB/s | 54 kB 00:00 2024-09-22T03:47:33.152 INFO:teuthology.orchestra.run.smithi083.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 3.7 MB/s | 220 kB 00:00 2024-09-22T03:47:33.177 INFO:teuthology.orchestra.run.smithi083.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-22T03:47:33.194 INFO:teuthology.orchestra.run.smithi083.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 5.3 MB/s | 90 kB 00:00 2024-09-22T03:47:33.219 INFO:teuthology.orchestra.run.smithi083.stdout:(12/13): python3-pyasn1-modules-0.4.8-6.el9.noa 584 kB/s | 279 kB 00:00 2024-09-22T03:47:33.270 INFO:teuthology.orchestra.run.smithi083.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-22T03:47:33.271 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:33.272 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.7 MB/s | 2.2 MB 00:00 2024-09-22T03:47:33.338 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:33.353 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:33.353 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:33.443 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:33.444 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:33.960 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:34.131 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:47:34.179 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:47:34.234 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:47:34.263 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:47:34.296 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:47:34.397 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:47:34.498 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:47:34.548 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:47:35.247 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:47:35.296 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:47:35.330 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:47:35.377 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:47:35.399 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:47:36.150 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:47:36.150 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-22T03:47:36.150 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:47:36.150 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:47:36.150 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:47:36.151 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:47:36.488 INFO:teuthology.orchestra.run.smithi083.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:36.489 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:36.759 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:47:37.339 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:01:58 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:37.473 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:37.473 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:37.473 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:37.474 INFO:teuthology.orchestra.run.smithi083.stdout:Install 6 Packages 2024-09-22T03:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 7.0 M 2024-09-22T03:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 33 M 2024-09-22T03:47:37.475 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:37.739 INFO:teuthology.orchestra.run.smithi083.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-22T03:47:37.805 INFO:teuthology.orchestra.run.smithi083.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-09-22T03:47:37.831 INFO:teuthology.orchestra.run.smithi083.stdout:(3/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 554 kB/s | 154 kB 00:00 2024-09-22T03:47:37.856 INFO:teuthology.orchestra.run.smithi083.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-22T03:47:37.890 INFO:teuthology.orchestra.run.smithi083.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-22T03:47:38.907 INFO:teuthology.orchestra.run.smithi083.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.4 MB/s | 6.0 MB 00:01 2024-09-22T03:47:38.909 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:38.909 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.9 MB/s | 7.0 MB 00:01 2024-09-22T03:47:38.979 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:38.991 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:38.991 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:39.092 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:39.093 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:39.416 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:39.580 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:47:39.634 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:47:40.151 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:47:40.245 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:47:40.313 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:47:40.338 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:47:41.176 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:47:41.451 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:41.452 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:41.741 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-fuse 2024-09-22T03:47:42.319 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:03 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:42.451 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:42.451 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout:Install 2 Packages 2024-09-22T03:47:42.452 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:42.453 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 963 k 2024-09-22T03:47:42.453 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 4.1 M 2024-09-22T03:47:42.453 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:42.759 INFO:teuthology.orchestra.run.smithi083.stdout:(1/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.3 MB/s | 883 kB 00:00 2024-09-22T03:47:42.918 INFO:teuthology.orchestra.run.smithi083.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 191 kB/s | 80 kB 00:00 2024-09-22T03:47:42.918 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:42.918 INFO:teuthology.orchestra.run.smithi083.stdout:Total 2.0 MB/s | 963 kB 00:00 2024-09-22T03:47:42.930 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:42.954 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:42.954 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:43.034 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:43.034 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:43.218 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:43.370 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:47:43.442 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:47:44.140 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:47:44.140 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:44.420 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:44.679 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install ceph-volume 2024-09-22T03:47:45.271 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:06 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:45.407 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:45.407 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 285 k 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 1.3 M 2024-09-22T03:47:45.408 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:45.631 INFO:teuthology.orchestra.run.smithi083.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-22T03:47:45.632 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:45.632 INFO:teuthology.orchestra.run.smithi083.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T03:47:45.632 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:45.640 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:45.641 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:45.667 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:45.668 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:45.885 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:45.949 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:45.966 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:45.967 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:47:45.967 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:47:45.967 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:47.004 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:47:47.004 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:47.004 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:47.004 INFO:teuthology.orchestra.run.smithi083.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:47:47.005 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:47.005 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:47.315 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados-devel 2024-09-22T03:47:47.901 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:08 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:48.034 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:48.035 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:48.035 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-22T03:47:48.035 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:48.035 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:48.035 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 125 k 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 445 k 2024-09-22T03:47:48.036 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:48.228 INFO:teuthology.orchestra.run.smithi083.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 649 kB/s | 125 kB 00:00 2024-09-22T03:47:48.229 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:48.229 INFO:teuthology.orchestra.run.smithi083.stdout:Total 643 kB/s | 125 kB 00:00 2024-09-22T03:47:48.229 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:48.235 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:48.235 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:48.298 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:48.299 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:48.447 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:48.550 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:49.035 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:49.319 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:49.320 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:49.320 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:49.320 INFO:teuthology.orchestra.run.smithi083.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:49.320 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:49.320 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:49.574 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs2 2024-09-22T03:47:50.167 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:11 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:50.256 INFO:teuthology.orchestra.run.smithi083.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:50.303 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:50.304 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:50.304 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:50.371 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install libcephfs-devel 2024-09-22T03:47:50.954 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:11 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:51.086 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:51.087 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:51.088 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 32 k 2024-09-22T03:47:51.088 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 142 k 2024-09-22T03:47:51.088 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:51.232 INFO:teuthology.orchestra.run.smithi083.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 220 kB/s | 32 kB 00:00 2024-09-22T03:47:51.233 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:51.233 INFO:teuthology.orchestra.run.smithi083.stdout:Total 218 kB/s | 32 kB 00:00 2024-09-22T03:47:51.233 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:51.237 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:51.237 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:51.249 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:51.249 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:51.338 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:51.463 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:51.855 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:52.110 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:52.346 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librados2 2024-09-22T03:47:52.928 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:13 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:53.017 INFO:teuthology.orchestra.run.smithi083.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:53.071 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:53.072 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:53.072 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:53.127 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install librbd1 2024-09-22T03:47:53.709 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:14 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:53.799 INFO:teuthology.orchestra.run.smithi083.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:53.841 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:53.842 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:53.843 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:53.918 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rados 2024-09-22T03:47:54.500 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:15 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:54.588 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:54.631 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:54.632 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:54.632 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:54.701 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rgw 2024-09-22T03:47:55.277 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:55.368 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:55.411 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:55.412 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:55.412 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:55.486 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-cephfs 2024-09-22T03:47:56.066 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:56.156 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:56.199 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:56.200 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:56.200 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:56.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-rbd 2024-09-22T03:47:56.842 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:56.931 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:47:56.974 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:56.975 INFO:teuthology.orchestra.run.smithi083.stdout:Nothing to do. 2024-09-22T03:47:56.975 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:57.054 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-fuse 2024-09-22T03:47:57.625 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:18 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:57.755 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:57.756 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:57.757 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 91 k 2024-09-22T03:47:57.757 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 321 k 2024-09-22T03:47:57.757 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:47:57.960 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 447 kB/s | 91 kB 00:00 2024-09-22T03:47:57.961 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:47:57.961 INFO:teuthology.orchestra.run.smithi083.stdout:Total 444 kB/s | 91 kB 00:00 2024-09-22T03:47:57.961 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:47:57.970 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:47:57.970 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:47:58.033 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:47:58.033 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:47:58.185 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:47:58.291 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:58.659 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:58.929 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:47:58.930 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:58.930 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:47:58.930 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:47:58.930 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:58.930 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:47:59.217 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-mirror 2024-09-22T03:47:59.818 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:20 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:47:59.963 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repo Size 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:47:59.964 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:47:59.965 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 3.5 M 2024-09-22T03:47:59.965 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 30 M 2024-09-22T03:47:59.965 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:48:00.324 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 9.8 MB/s | 3.5 MB 00:00 2024-09-22T03:48:00.324 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:48:00.325 INFO:teuthology.orchestra.run.smithi083.stdout:Total 9.7 MB/s | 3.5 MB 00:00 2024-09-22T03:48:00.325 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:48:00.336 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:48:00.336 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:48:00.420 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:48:00.420 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:48:01.002 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:48:01.066 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:48:01.092 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:02.318 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:48:02.595 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install rbd-nbd 2024-09-22T03:48:03.187 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:02:24 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:48:03.322 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout: Package Architecture Version Repository Size 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:03.323 INFO:teuthology.orchestra.run.smithi083.stdout:Install 1 Package 2024-09-22T03:48:03.324 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:03.324 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 176 k 2024-09-22T03:48:03.324 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 685 k 2024-09-22T03:48:03.324 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:48:03.595 INFO:teuthology.orchestra.run.smithi083.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 651 kB/s | 176 kB 00:00 2024-09-22T03:48:03.596 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:48:03.596 INFO:teuthology.orchestra.run.smithi083.stdout:Total 648 kB/s | 176 kB 00:00 2024-09-22T03:48:03.596 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:48:03.605 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:48:03.605 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:48:03.670 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:48:03.670 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:48:03.830 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:48:03.935 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:04.377 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:04.617 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:48:04.618 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:04.618 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:48:04.618 INFO:teuthology.orchestra.run.smithi083.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:48:04.618 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:04.618 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:48:04.874 DEBUG:teuthology.parallel:result is None 2024-09-22T03:48:04.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:48:05.017 DEBUG:teuthology.orchestra.run.smithi083:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:48:05.037 INFO:teuthology.orchestra.run.smithi083.stdout:19.3.0-5092.g62da54b4.el9 2024-09-22T03:48:05.038 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-22T03:48:05.038 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-22T03:48:05.039 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:48:05.039 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:05.040 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:48:05.102 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:48:05.102 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:05.102 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:48:05.168 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:48:05.233 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:48:05.233 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:05.233 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:48:05.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:48:05.362 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:48:05.362 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:05.363 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:48:05.427 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:48:05.492 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:48:05.566 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:48:05.566 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 777 /var/log/ceph 2024-09-22T03:48:05.591 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:48:05.591 DEBUG:teuthology.orchestra.run.smithi083:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:48:05.667 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:48:05.667 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:48:05.746 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:48:05.746 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:48:05.747 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914376', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/fio_4M_rand_read}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914376', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi083.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDouoPibb/w941bFOT2pFmswA0CLROENd3gZt+9bAAXjEijlKBgloJcGS8bpKBXmyd/BED1UtYXw/14ynAzBdjU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:48:05.747 DEBUG:teuthology.orchestra.run.smithi083:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:48:05.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:48:05.851 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:05.851 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:48:05.907 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:48:05.907 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_1 2024-09-22T03:48:05.962 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:48:05.962 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:48:05.962 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-22 03:48:01.341901668 +0000 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-22 03:45:01.940848783 +0000 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-22 03:45:01.940848783 +0000 2024-09-22T03:48:05.963 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-22 03:45:01.940848783 +0000 2024-09-22T03:48:05.963 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:48:06.025 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-22T03:48:06.025 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-22T03:48:06.025 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000207796 s, 2.5 MB/s 2024-09-22T03:48:06.026 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:48:06.082 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_2 2024-09-22T03:48:06.137 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:48:06.137 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-22 03:48:01.341901668 +0000 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-22 03:45:02.268840859 +0000 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-22 03:45:02.268840859 +0000 2024-09-22T03:48:06.138 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-22 03:45:02.268840859 +0000 2024-09-22T03:48:06.138 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:48:06.200 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-22T03:48:06.200 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-22T03:48:06.200 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000193425 s, 2.6 MB/s 2024-09-22T03:48:06.202 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:48:06.257 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_3 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-22 03:48:01.341901668 +0000 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-22 03:45:02.591833055 +0000 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-22 03:45:02.591833055 +0000 2024-09-22T03:48:06.313 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-22 03:45:02.591833055 +0000 2024-09-22T03:48:06.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:48:06.376 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-22T03:48:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-22T03:48:06.377 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.000197718 s, 2.6 MB/s 2024-09-22T03:48:06.378 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:48:06.434 DEBUG:teuthology.orchestra.run.smithi083:> stat /dev/vg_nvme/lv_4 2024-09-22T03:48:06.489 INFO:teuthology.orchestra.run.smithi083.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:48:06.489 INFO:teuthology.orchestra.run.smithi083.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Access: 2024-09-22 03:48:01.342901644 +0000 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Modify: 2024-09-22 03:45:02.935824745 +0000 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout:Change: 2024-09-22 03:45:02.935824745 +0000 2024-09-22T03:48:06.490 INFO:teuthology.orchestra.run.smithi083.stdout: Birth: 2024-09-22 03:45:02.935824745 +0000 2024-09-22T03:48:06.490 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:48:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records in 2024-09-22T03:48:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:1+0 records out 2024-09-22T03:48:06.553 INFO:teuthology.orchestra.run.smithi083.stderr:512 bytes copied, 0.00020942 s, 2.4 MB/s 2024-09-22T03:48:06.554 DEBUG:teuthology.orchestra.run.smithi083:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:48:06.610 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:48:06.611 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:48:06.611 INFO:tasks.ceph:Generating config... 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-22T03:48:06.612 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-22T03:48:06.613 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-22T03:48:06.614 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-22T03:48:06.614 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-22T03:48:06.614 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:48:06.614 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T03:48:06.614 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T03:48:06.615 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T03:48:06.616 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T03:48:06.617 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:48:06.618 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:48:06.619 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T03:48:06.619 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:48:06.619 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:48:06.619 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:48:06.619 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:48:06.619 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:48:06.619 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:48:06.620 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:48:06.705 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:48:06.709 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:48:06.762 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:48:06.786 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.83')] 2024-09-22T03:48:06.786 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.83', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:48:06.787 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:06.787 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:48:06.842 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.83 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:06.932 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:06.930+0000 7fb50eae1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:06.930+0000 7fb50eae1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:06.931+0000 7fb50eae1f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: generated fsid fa2b2c8b-7e2f-44e7-8f74-285b20a4f719 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:setting min_mon_release = quincy 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:epoch 0 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:fsid fa2b2c8b-7e2f-44e7-8f74-285b20a4f719 2024-09-22T03:48:06.933 INFO:teuthology.orchestra.run.smithi083.stdout:last_changed 2024-09-22T03:48:06.932749+0000 2024-09-22T03:48:06.934 INFO:teuthology.orchestra.run.smithi083.stdout:created 2024-09-22T03:48:06.932749+0000 2024-09-22T03:48:06.934 INFO:teuthology.orchestra.run.smithi083.stdout:min_mon_release 17 (quincy) 2024-09-22T03:48:06.934 INFO:teuthology.orchestra.run.smithi083.stdout:election_strategy: 1 2024-09-22T03:48:06.934 INFO:teuthology.orchestra.run.smithi083.stdout:0: [v2:172.21.15.83:3300/0,v1:172.21.15.83:6789/0] mon.a 2024-09-22T03:48:06.934 INFO:teuthology.orchestra.run.smithi083.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:48:06.935 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:48:06.949 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fa2b2c8b-7e2f-44e7-8f74-285b20a4f719... 2024-09-22T03:48:06.951 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:48:07.032 INFO:teuthology.orchestra.run.smithi083.stdout:[global] 2024-09-22T03:48:07.032 INFO:teuthology.orchestra.run.smithi083.stdout: chdir = "" 2024-09-22T03:48:07.032 INFO:teuthology.orchestra.run.smithi083.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:48:07.032 INFO:teuthology.orchestra.run.smithi083.stdout: auth supported = none 2024-09-22T03:48:07.032 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: filestore xattr use omap = true 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: mon clock drift allowed = 1.000 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: osd crush chooseleaf type = 0 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: auth debug = true 2024-09-22T03:48:07.033 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on old message = true 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: ms die on bug = true 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: mon pg warn max object skew = 0 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default size = 2 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.034 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow primary affinity = true 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd allow pg remap = true 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on no sortbitwise = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon warn on too few osds = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: mon_allow_pool_size_one = true 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: osd default data pool replay window = 5 2024-09-22T03:48:07.035 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: mon allow pool delete = true 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: mon cluster log file level = debug 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: debug asserts on shutdown = true 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: mon health detail to clog = false 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: mon host = 172.21.15.83 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: auth client required = none 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: auth cluster required = none 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: auth service required = none 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: debug asok = 0/0 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 0/0 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: debug bdev = 0/0 2024-09-22T03:48:07.036 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluefs = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug bluestore = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug buffer = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug civetweb = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug client = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug compressor = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug context = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug crush = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug crypto = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug dpdk = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug eventtrace = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug filer = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug filestore = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug finisher = 0/0 2024-09-22T03:48:07.037 INFO:teuthology.orchestra.run.smithi083.stdout: debug fuse = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug heartbeatmap = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug javaclient = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug journal = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug journaler = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug kstore = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug leveldb = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug lockdep = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds balancer = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds locker = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds log = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds log expire = 0/0 2024-09-22T03:48:07.038 INFO:teuthology.orchestra.run.smithi083.stdout: debug mds migrator = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug memdb = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgrc = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug objclass = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug objectcacher = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug objecter = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug optracker = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 0/0 2024-09-22T03:48:07.039 INFO:teuthology.orchestra.run.smithi083.stdout: debug perfcounter = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rados = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd mirror = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rbd replay = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug refs = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw lifecycle = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw notification = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rgw sync = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug rocksdb = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug striper = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug throttle = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug timer = 0/0 2024-09-22T03:48:07.040 INFO:teuthology.orchestra.run.smithi083.stdout: debug tp = 0/0 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms client mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms cluster mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon client mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon cluster mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms mon service mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: ms service mode = crc 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: fsid = fa2b2c8b-7e2f-44e7-8f74-285b20a4f719 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout:[osd] 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: osd journal size = 100 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub load threshold = 5.0 2024-09-22T03:48:07.041 INFO:teuthology.orchestra.run.smithi083.stdout: osd scrub max interval = 600 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd recover clone overlap = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd recovery max chunk = 1048576 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug shutdown = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug op order = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug verify stray on activate = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug trim objects = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd open classes on start = true 2024-09-22T03:48:07.042 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug pg log writeout = true 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: osd map max advance = 10 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: journal zero on create = true 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: filestore apply finisher threads = 3 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: bdev debug aio = true 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: osd debug misdirected ops = true 2024-09-22T03:48:07.043 INFO:teuthology.orchestra.run.smithi083.stdout: debug monc = 20 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: crimson osd obc lru size = 10 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug filestore = 0/0 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug journal = 0/0 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 0/0 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug osd = 0/0 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore = 20 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore backref = 20 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore cache = 20 2024-09-22T03:48:07.044 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore cleaner = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore device = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore epm = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore fixedkv tree = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore journal = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore lba = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore odata = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore ompap = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore onode = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore t = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: debug seastore tm = 20 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: osd objectstore = seastore 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: osd shutdown pgref assert = True 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.045 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr] 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 1 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug mgr = 20 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 20 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: mgr/telemetry/nag = false 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout:[mon] 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug ms = 0/0 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug mon = 0/0 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug paxos = 0/0 2024-09-22T03:48:07.046 INFO:teuthology.orchestra.run.smithi083.stdout: debug auth = 20 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon data avail warn = 5 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon mgr mkfs grace = 240 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd reporter subtree level = osd 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon osd prime pg temp = true 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.047 INFO:teuthology.orchestra.run.smithi083.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: # 1m isn't quite enough 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: osd pool default crimson = True 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout:[client] 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: rgw cache enabled = true 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable ops log = true 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: rgw enable usage log = true 2024-09-22T03:48:07.048 INFO:teuthology.orchestra.run.smithi083.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout:[mon.a] 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.3] 2024-09-22T03:48:07.049 INFO:teuthology.orchestra.run.smithi083.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:48:07.050 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:48:07.050 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:48:07.156 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:48:07.156 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:07.156 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:48:07.171 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:07.172 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:48:07.227 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi083.front.sepia.ceph.com 2024-09-22T03:48:07.227 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:07.227 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:48:07.228 DEBUG:teuthology.orchestra.run.smithi083:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:48:07.301 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:07.301 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:07.356 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:48:07.357 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:48:07.357 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:48:07.455 INFO:teuthology.orchestra.run.smithi083.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:48:07.458 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:48:07.458 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:48:07.459 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:48:07.508 INFO:teuthology.orchestra.run.smithi083.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:48:07.521 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:48:07.521 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi083.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:48:07.521 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:48:07.545 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:48:07.545 INFO:tasks.ceph:role: osd.0 2024-09-22T03:48:07.545 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-22T03:48:07.545 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:48:07.617 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:48:07.618 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:48:07.618 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:48:08.135 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-22T03:48:08.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-22T03:48:08.137 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:48:08.272 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:48:08.303 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:48:08.368 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:48:08.369 INFO:tasks.ceph:role: osd.1 2024-09-22T03:48:08.369 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-22T03:48:08.369 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:48:08.441 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:48:08.441 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:48:08.441 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:48:08.441 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:48:08.442 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:48:08.889 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-22T03:48:08.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-22T03:48:08.891 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:48:08.979 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:48:09.009 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:48:09.075 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:48:09.075 INFO:tasks.ceph:role: osd.2 2024-09-22T03:48:09.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com 2024-09-22T03:48:09.076 DEBUG:teuthology.orchestra.run.smithi083:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:48:09.148 INFO:teuthology.orchestra.run.smithi083.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:48:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:48:09.148 INFO:teuthology.orchestra.run.smithi083.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:48:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:48:09.149 INFO:teuthology.orchestra.run.smithi083.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:48:09.149 INFO:teuthology.orchestra.run.smithi083.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:48:09.149 INFO:teuthology.orchestra.run.smithi083.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:48:09.149 INFO:teuthology.orchestra.run.smithi083.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:48:09.150 INFO:teuthology.orchestra.run.smithi083.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:48:09.150 INFO:teuthology.orchestra.run.smithi083.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:48:09.639 INFO:teuthology.orchestra.run.smithi083.stdout:Discarding blocks...Done. 2024-09-22T03:48:09.641 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi083.front.sepia.ceph.com -o noatime 2024-09-22T03:48:09.641 DEBUG:teuthology.orchestra.run.smithi083:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:48:09.795 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:48:09.828 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:10.133 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:10.134 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:10,134 seastar - Reactor backend: linux-aio 2024-09-22T03:48:10.149 INFO:teuthology.orchestra.run.smithi083.stderr:==28202==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:10.278 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-22 03:48:10,277 [shard 0:main] seastar - reactor::drain 2024-09-22T03:48:10.786 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:10.787 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:10.787 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:10.787 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:10.787 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:10.844 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:10.848 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:10,848 seastar - Reactor backend: linux-aio 2024-09-22T03:48:10.857 INFO:teuthology.orchestra.run.smithi083.stderr:==28200==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x36074ec 0x3b8bf66 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c2b 0xdd332 0x4ab1c 0xb9cc4a3 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.140 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x377977e 0x6b6618d 0x6b66481 0x365580c 0x3655a44 0x36dcf03 0x36dd2e0 0x36e08d7 0x36e0d04 0x36e11ca 0x36e1614 0xc4a9be1 0xc494e6d 0xc495542 0xb0b6299 0xb0b0970 0xac78421 0xacde0f1 0xacab3b6 0x99296ea 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd32a 0x4ab1c 0x5fccc96 0x5fcce78 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5df61a3 0x5df7058 0x4efe599 0x4efe84c 0x957652e 0xac78378 0xacde0f1 0xacab3b6 0x99296ea 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.198 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525dba 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:11.403 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x552518c 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:11.460 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525816 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.483 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.485 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.485 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253bc 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.541 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.543 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b56c 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.543 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.568 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab2c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:11.568 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.842 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156f9 0xdd312 0x4ab1c 0x3ac7302 0x3b8c153 0xb9d336b 0xb9d6dd9 0xba1a7fc 0xb9bba82 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.842 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3ad43 0x10d76c 0xdd31a 0x4b14c 0xb9cdba8 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:11.842 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.842 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:11.976 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525fe5 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:11.976 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:12.033 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55251ba 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:12.034 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:12.919 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:12.919 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:12.920 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:12.920 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:12.920 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:12.920 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:12.946 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:48:12.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:13.248 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:13.250 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:13,249 seastar - Reactor backend: linux-aio 2024-09-22T03:48:13.255 INFO:teuthology.orchestra.run.smithi083.stderr:==28249==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:13.384 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-22 03:48:13,384 [shard 0:main] seastar - reactor::drain 2024-09-22T03:48:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:13.891 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:13.892 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:13.892 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:13.892 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:13.892 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:13.945 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:13.949 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:13,949 seastar - Reactor backend: linux-aio 2024-09-22T03:48:13.962 INFO:teuthology.orchestra.run.smithi083.stderr:==28247==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:14.247 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797f 0xdd2fc 0x4ab1c 0x3ac7302 0x3b8c153 0xb9d39f8 0xb9bb9f1 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.247 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xfe578 0x5ca6522 0x5ca66fc 0x5ca69b3 0x5ca6aa8 0xb9ffe54 0xb9bfd6e 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.247 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.247 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.305 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x117f8f 0x60cd8e4 0x9eb3dc5 0x9eb457c 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.305 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.305 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4bebf 0x9dea483 0x9df281b 0x9e0571e 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.305 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.306 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a1e5 0xfb55e 0x3767e9b 0x5ca664d 0x5ca69b3 0x5ca6aa8 0xb9ffe54 0xba0b7a4 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:14.306 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc9199 0xc9e39 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9e53f4 0xba14771 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:14.363 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.494 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253e4 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:14.495 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.552 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5524ead 0x5525e58 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:14.552 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.585 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525e71 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.585 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.586 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525767 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.586 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.643 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab49 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.643 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.644 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.644 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.659 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ca9 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:14.659 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.959 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8bf05 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.959 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3779709 0x3b8beba 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:14.959 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:14.959 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:16.037 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:16.063 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:48:16.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:16.372 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:16.373 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:16,373 seastar - Reactor backend: linux-aio 2024-09-22T03:48:16.379 INFO:teuthology.orchestra.run.smithi083.stderr:==28296==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:16.508 INFO:teuthology.orchestra.run.smithi083.stderr:DEBUG 2024-09-22 03:48:16,507 [shard 0:main] seastar - reactor::drain 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:17.007 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:17.064 INFO:teuthology.orchestra.run.smithi083.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:48:17.068 INFO:teuthology.orchestra.run.smithi083.stderr:INFO 2024-09-22 03:48:17,068 seastar - Reactor backend: linux-aio 2024-09-22T03:48:17.088 INFO:teuthology.orchestra.run.smithi083.stderr:==28294==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:48:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xf9de8 0x3868e75 0x386922f 0xba0df79 0xba0e13c 0xba0e2a9 0xb9b5385 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x38192eb 0x5bf7429 0x5bf7621 0x5c2529f 0xb8382d8 0xba09156 0xb9ae311 0x9adcd51 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.373 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.431 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x363e571 0xba1712b 0xba17729 0xb9b22dd 0x9826b31 0x9df4b97 0x9e0571e 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.431 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.431 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x76be046 0xba0361e 0xba0e0dd 0xba0e2a9 0xb9b5385 0xb9b58b4 0xb9bd31f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9df53d6 0x9e0571e 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.431 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.431 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb836b07 0xb837196 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xb9bdf19 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:17.432 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3868fe1 0xb83639c 0xb83655a 0xb836643 0xb836903 0xba0a005 0xba0a0f4 0xba0a7b4 0xba0a91f 0xba0ade1 0xba0b239 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:17.489 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab61 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:17.619 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.676 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10d778 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:17.677 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.691 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cf0 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.691 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x53b10 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.723 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.748 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.748 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.781 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abef 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.781 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133cca 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:48:17.784 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 197 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525829 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:17.856 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 2024-09-22T03:48:18.085 INFO:teuthology.orchestra.run.smithi083.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4218a 0x3ecea 0xfc9b1 0x3869141 0xba09592 0xba0b737 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:48:18.085 INFO:teuthology.orchestra.run.smithi083.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa5d96768 0xffffffffa5d96e1c 0xffffffffa5a8c884 0xffffffffa66d1db2 0xffffffffa6800bb2 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr:Suppressions used: 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr: count bytes template 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr: 1 8 MallocExtension::Register 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr:----------------------------------------------------- 2024-09-22T03:48:19.198 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:19.225 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:48:19.249 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:48:19.249 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:19.249 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:48:19.314 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:19.314 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:48:19.377 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:19.378 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:48:19.441 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:19.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:48:19.506 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:19.506 DEBUG:teuthology.orchestra.run.smithi083:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:48:19.560 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:48:19.561 DEBUG:teuthology.orchestra.run.smithi083:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:48:19.623 INFO:teuthology.orchestra.run.smithi083.stdout:[mgr.x] 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB3k+9md68kGxAABbabEfyLf5b2tOu0z/DhEw== 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.0] 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB6k+9mlyuPOxAA6i45R3sm4jsc03zJe8lnSQ== 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.1] 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB+k+9mH01dBhAALoElkZd2qOCHaxOH5GzQiw== 2024-09-22T03:48:19.624 INFO:teuthology.orchestra.run.smithi083.stdout:[osd.2] 2024-09-22T03:48:19.625 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQCBk+9mMN0NDhAAE3cUZKJVGkEo7zrqzraZQw== 2024-09-22T03:48:19.625 INFO:teuthology.orchestra.run.smithi083.stdout:[client.0] 2024-09-22T03:48:19.625 INFO:teuthology.orchestra.run.smithi083.stdout: key = AQB3k+9mK9dHHhAAAv9/GfejkOlvM/mPpRY+Cg== 2024-09-22T03:48:19.625 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:48:19.726 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:48:19.786 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:48:19.847 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:48:19.908 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:48:19.972 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:48:19.972 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:48:19.996 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:48:20.136 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.135+0000 7f8dcda2dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.137 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.135+0000 7f8dcda2dc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.149 DEBUG:teuthology.orchestra.run.smithi083:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:48:20.173 DEBUG:teuthology.orchestra.run.smithi083:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:48:20.228 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:48:20.229 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:48:20.229 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:48:20.271 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:48:20.271 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:48:20.271 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:48:20.271 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:48:20.273 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:48:20.273 DEBUG:tasks.ceph:set 0 configs 2024-09-22T03:48:20.273 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:48:20.273 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:48:20.353 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:48:20.351+0000 7fdc738f7c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.353 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:48:20.351+0000 7fdc738f7c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.354 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:48:20.352+0000 7fdc738f7c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.354 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-22T03:48:20.352+0000 7f5c8d6ff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.382 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.375+0000 7f3e59046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.386 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.385+0000 7f3e59046640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.455 INFO:teuthology.orchestra.run.smithi083.stderr:adjusted tunables profile to default 2024-09-22T03:48:20.466 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:48:20.466 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:48:20.466 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:48:20.558 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-22T03:48:20.556+0000 7f5c8d6ff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.558 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-22T03:48:20.556+0000 7f5c8d6ff000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.575 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.569+0000 7f0928391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.577 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:20.576+0000 7f0928391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:20.654 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:48:20.653+0000 7fdc6a5f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:48:20.655 INFO:tasks.ceph.mon.a.smithi083.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:48:20.655 INFO:tasks.ceph.mon.a.smithi083.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:48:20.655 INFO:tasks.ceph.mon.a.smithi083.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:48:20.655 INFO:tasks.ceph.mon.a.smithi083.stderr:feature with important data. 2024-09-22T03:48:20.655 INFO:tasks.ceph.mon.a.smithi083.stderr: 2024-09-22T03:48:21.467 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:48:21.468 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:21.468 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:48:21.492 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:21.492 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:48:21.557 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:21.557 DEBUG:teuthology.orchestra.run.smithi083:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:48:21.622 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 3e669f99-197d-4c7d-a288-3d38c3a7975e 0 2024-09-22T03:48:21.770 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:21.763+0000 7f12a6ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:21.772 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:21.771+0000 7f12a6ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:21.848 INFO:teuthology.orchestra.run.smithi083.stdout:0 2024-09-22T03:48:21.859 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 64d3b1e5-0f39-44f6-bc38-953c5006add3 1 2024-09-22T03:48:21.966 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:21.959+0000 7f26bf0c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:21.968 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:21.967+0000 7f26bf0c8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:22.044 INFO:teuthology.orchestra.run.smithi083.stdout:1 2024-09-22T03:48:22.056 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd new 74b00a93-bdf3-4924-bf06-cb81bf3edab9 2 2024-09-22T03:48:22.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:22.156+0000 7f2342c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:22.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:22.164+0000 7f2342c74640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:22.256 INFO:teuthology.orchestra.run.smithi083.stdout:2 2024-09-22T03:48:22.268 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:48:22.298 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:48:22.298 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:48:22.340 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:48:22.340 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:48:22.369 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:48:22.369 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:48:22.412 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:48:22.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:48:22.442 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:48:22.442 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:48:22.485 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:48:22.485 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:48:22.647 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:22.641+0000 7f5586274640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:22.649 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:22.648+0000 7f5586274640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:22.719 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:22.719 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":6,"fsid":"fa2b2c8b-7e2f-44e7-8f74-285b20a4f719","created":"2024-09-22T03:48:20.379843+0000","modified":"2024-09-22T03:48:22.255666+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:48:22.255666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e669f99-197d-4c7d-a288-3d38c3a7975e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"64d3b1e5-0f39-44f6-bc38-953c5006add3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74b00a93-bdf3-4924-bf06-cb81bf3edab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:48:22.731 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:48:22.731 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:48:23.033 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:48:23.136 INFO:teuthology.misc.health.smithi083.stderr:2024-09-22T03:48:23.130+0000 7f7b80391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:23.138 INFO:teuthology.misc.health.smithi083.stderr:2024-09-22T03:48:23.137+0000 7f7b80391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:23.205 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-22T03:48:23.206 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":6,"fsid":"fa2b2c8b-7e2f-44e7-8f74-285b20a4f719","created":"2024-09-22T03:48:20.379843+0000","modified":"2024-09-22T03:48:22.255666+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:48:22.255666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e669f99-197d-4c7d-a288-3d38c3a7975e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"64d3b1e5-0f39-44f6-bc38-953c5006add3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74b00a93-bdf3-4924-bf06-cb81bf3edab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:48:23.216 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:48:25.944 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-22T03:48:25.943+0000 7f5c2aec4640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:48:29.519 DEBUG:teuthology.orchestra.run.smithi083:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:48:29.627 INFO:teuthology.misc.health.smithi083.stderr:2024-09-22T03:48:29.622+0000 7f6c22274640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:29.630 INFO:teuthology.misc.health.smithi083.stderr:2024-09-22T03:48:29.629+0000 7f6c22274640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:30.025 INFO:teuthology.misc.health.smithi083.stdout: 2024-09-22T03:48:30.026 INFO:teuthology.misc.health.smithi083.stdout:{"epoch":10,"fsid":"fa2b2c8b-7e2f-44e7-8f74-285b20a4f719","created":"2024-09-22T03:48:20.379843+0000","modified":"2024-09-22T03:48:29.002132+0000","last_up_change":"2024-09-22T03:48:29.002132+0000","last_in_change":"2024-09-22T03:48:22.255666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"3e669f99-197d-4c7d-a288-3d38c3a7975e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":4179963007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":4179963007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":4179963007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":4179963007}]},"public_addr":"172.21.15.83:6808/4179963007","cluster_addr":"172.21.15.83:6807/4179963007","heartbeat_back_addr":"172.21.15.83:6815/4179963007","heartbeat_front_addr":"172.21.15.83:6811/4179963007","state":["exists","up"]},{"osd":1,"uuid":"64d3b1e5-0f39-44f6-bc38-953c5006add3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":4026345099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":4026345099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":4026345099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":4026345099}]},"public_addr":"172.21.15.83:6803/4026345099","cluster_addr":"172.21.15.83:6802/4026345099","heartbeat_back_addr":"172.21.15.83:6805/4026345099","heartbeat_front_addr":"172.21.15.83:6804/4026345099","state":["exists","up"]},{"osd":2,"uuid":"74b00a93-bdf3-4924-bf06-cb81bf3edab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1367698956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1367698956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1367698956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6813","nonce":1367698956}]},"public_addr":"172.21.15.83:6810/1367698956","cluster_addr":"172.21.15.83:6809/1367698956","heartbeat_back_addr":"172.21.15.83:6814/1367698956","heartbeat_front_addr":"172.21.15.83:6813/1367698956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:48:30.045 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:48:30.045 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:48:30.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:48:30.164 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:30.155+0000 7fa79ad51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:30.166 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:30.166+0000 7fa79ad51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:31.011 INFO:teuthology.orchestra.run.smithi083.stderr:pool 'rbd' created 2024-09-22T03:48:31.038 DEBUG:teuthology.orchestra.run.smithi083:> rbd --cluster ceph pool init rbd 2024-09-22T03:48:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:31.150+0000 7fe160741ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:31.151 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:31.150+0000 7fe160741ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:31.154 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:31.154+0000 7fe160741ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.087 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:48:34.087 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-22T03:48:34.088 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:48:34.200 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.195+0000 7fa068791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.209 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.202+0000 7fa068791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.393 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:34.405 INFO:teuthology.orchestra.run.smithi083.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-22T03:48:34.405 INFO:tasks.ceph_manager:config epoch is 1 2024-09-22T03:48:34.406 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:48:34.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:48:34.406 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:48:34.514 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.510+0000 7f8ec1791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.516 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.516+0000 7f8ec1791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.755 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:34.766 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6800","nonce":932598297},{"type":"v1","addr":"172.21.15.83:6801","nonce":932598297}]},"active_addr":"172.21.15.83:6801/932598297","active_change":"2024-09-22T03:48:24.932407+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":198622661}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":3930601760}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.83:0","nonce":587386429}]}]} 2024-09-22T03:48:34.771 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:48:34.771 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:48:34.772 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:48:34.918 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.914+0000 7fbd27d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:34.920 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:34.920+0000 7fbd27d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.108 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:35.108 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"fsid":"fa2b2c8b-7e2f-44e7-8f74-285b20a4f719","created":"2024-09-22T03:48:20.379843+0000","modified":"2024-09-22T03:48:34.017708+0000","last_up_change":"2024-09-22T03:48:29.002132+0000","last_in_change":"2024-09-22T03:48:22.255666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:48:30.372507+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:48:30.955457+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3e669f99-197d-4c7d-a288-3d38c3a7975e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":4179963007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":4179963007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":4179963007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":4179963007}]},"public_addr":"172.21.15.83:6808/4179963007","cluster_addr":"172.21.15.83:6807/4179963007","heartbeat_back_addr":"172.21.15.83:6815/4179963007","heartbeat_front_addr":"172.21.15.83:6811/4179963007","state":["exists","up"]},{"osd":1,"uuid":"64d3b1e5-0f39-44f6-bc38-953c5006add3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":4026345099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":4026345099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":4026345099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":4026345099}]},"public_addr":"172.21.15.83:6803/4026345099","cluster_addr":"172.21.15.83:6802/4026345099","heartbeat_back_addr":"172.21.15.83:6805/4026345099","heartbeat_front_addr":"172.21.15.83:6804/4026345099","state":["exists","up"]},{"osd":2,"uuid":"74b00a93-bdf3-4924-bf06-cb81bf3edab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1367698956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1367698956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1367698956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6813","nonce":1367698956}]},"public_addr":"172.21.15.83:6810/1367698956","cluster_addr":"172.21.15.83:6809/1367698956","heartbeat_back_addr":"172.21.15.83:6814/1367698956","heartbeat_front_addr":"172.21.15.83:6813/1367698956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:48:35.121 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:48:35.121 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:48:35.232 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.228+0000 7fe36d2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.234 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.234+0000 7fe36d2d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.422 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:35.422 INFO:teuthology.orchestra.run.smithi083.stdout:{"epoch":14,"fsid":"fa2b2c8b-7e2f-44e7-8f74-285b20a4f719","created":"2024-09-22T03:48:20.379843+0000","modified":"2024-09-22T03:48:34.017708+0000","last_up_change":"2024-09-22T03:48:29.002132+0000","last_in_change":"2024-09-22T03:48:22.255666+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:48:30.372507+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:48:30.955457+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"3e669f99-197d-4c7d-a288-3d38c3a7975e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6808","nonce":4179963007}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6807","nonce":4179963007}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6815","nonce":4179963007}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6811","nonce":4179963007}]},"public_addr":"172.21.15.83:6808/4179963007","cluster_addr":"172.21.15.83:6807/4179963007","heartbeat_back_addr":"172.21.15.83:6815/4179963007","heartbeat_front_addr":"172.21.15.83:6811/4179963007","state":["exists","up"]},{"osd":1,"uuid":"64d3b1e5-0f39-44f6-bc38-953c5006add3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6803","nonce":4026345099}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6802","nonce":4026345099}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6805","nonce":4026345099}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6804","nonce":4026345099}]},"public_addr":"172.21.15.83:6803/4026345099","cluster_addr":"172.21.15.83:6802/4026345099","heartbeat_back_addr":"172.21.15.83:6805/4026345099","heartbeat_front_addr":"172.21.15.83:6804/4026345099","state":["exists","up"]},{"osd":2,"uuid":"74b00a93-bdf3-4924-bf06-cb81bf3edab9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6810","nonce":1367698956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6809","nonce":1367698956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6814","nonce":1367698956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.83:6813","nonce":1367698956}]},"public_addr":"172.21.15.83:6810/1367698956","cluster_addr":"172.21.15.83:6809/1367698956","heartbeat_back_addr":"172.21.15.83:6814/1367698956","heartbeat_front_addr":"172.21.15.83:6813/1367698956","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:48:35.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:48:35.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:48:35.436 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:48:35.550 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.545+0000 7f4cacf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.545+0000 7f825e111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.551 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.545+0000 7f2e4e191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.552 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.552+0000 7f4cacf51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.552+0000 7f2e4e191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.553 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.553+0000 7f825e111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.580 INFO:teuthology.orchestra.run.smithi083.stdout:34359738376 2024-09-22T03:48:35.581 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:48:35.582 INFO:teuthology.orchestra.run.smithi083.stdout:42949672966 2024-09-22T03:48:35.582 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:48:35.585 INFO:teuthology.orchestra.run.smithi083.stdout:42949672966 2024-09-22T03:48:35.585 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:48:35.696 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.690+0000 7f5c38246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.699 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.698+0000 7f5c38246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.734+0000 7fb3cf791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.739 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.734+0000 7f6fc6663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.741+0000 7fb3cf791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.742 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:35.742+0000 7f6fc6663640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:35.910 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-09-22T03:48:35.923 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T03:48:35.939 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:35.952 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:48:35.957 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:35.970 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:48:36.925 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:48:36.953 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:48:36.971 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:48:37.045 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.038+0000 7ff583693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.047 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.047+0000 7ff583693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.090 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.083+0000 7fd11f111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.092 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.092+0000 7fd11f111640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.126 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.120+0000 7fc46a463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.129 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:37.128+0000 7fc46a463640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:37.290 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-09-22T03:48:37.303 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T03:48:37.325 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:37.326 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:37.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:48:37.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:48:38.304 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:48:38.339 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:48:38.341 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:48:38.430 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.422+0000 7fb212391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.432 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.432+0000 7fb212391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.467 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.461+0000 7fcbe609c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.469 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.468+0000 7fcbe609c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.475 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.469+0000 7f6392191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.483 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:38.482+0000 7f6392191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:38.648 INFO:teuthology.orchestra.run.smithi083.stdout:34359738372 2024-09-22T03:48:38.661 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T03:48:38.670 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:38.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:48:38.684 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:38.696 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:48:39.662 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:48:39.684 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:48:39.697 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:48:39.792 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.783+0000 7fd929d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:39.794 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.793+0000 7fd929d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:39.826 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.820+0000 7fdec6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:39.829 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.827+0000 7fdec6b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:39.831 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.823+0000 7fa57feac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:39.842 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:39.841+0000 7fa57feac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:40.006 INFO:teuthology.orchestra.run.smithi083.stdout:34359738377 2024-09-22T03:48:40.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738377 for osd.1 2024-09-22T03:48:40.020 DEBUG:teuthology.parallel:result is None 2024-09-22T03:48:40.031 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:40.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:48:40.044 INFO:teuthology.orchestra.run.smithi083.stdout:42949672964 2024-09-22T03:48:40.057 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:48:41.045 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:48:41.057 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:48:41.159 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.152+0000 7fb16d4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.161 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.160+0000 7fb16d4fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.204 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.198+0000 7fc852551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.206 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.204+0000 7fc852551640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.361 INFO:teuthology.orchestra.run.smithi083.stdout:42949672969 2024-09-22T03:48:41.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.2 2024-09-22T03:48:41.375 DEBUG:teuthology.parallel:result is None 2024-09-22T03:48:41.398 INFO:teuthology.orchestra.run.smithi083.stdout:42949672969 2024-09-22T03:48:41.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.0 2024-09-22T03:48:41.411 DEBUG:teuthology.parallel:result is None 2024-09-22T03:48:41.411 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:48:41.412 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:48:41.560 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.555+0000 7f9f48191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.561 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.560+0000 7f9f48191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.747 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:41.748 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-22T03:48:41.760 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T03:48:40.938960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.799687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044404+0000","last_change":"2024-09-22T03:48:34.050664+0000","last_active":"2024-09-22T03:48:34.044404+0000","last_peered":"2024-09-22T03:48:34.044404+0000","last_clean":"2024-09-22T03:48:34.044404+0000","last_became_active":"2024-09-22T03:48:32.164891+0000","last_became_peered":"2024-09-22T03:48:32.164891+0000","last_unstale":"2024-09-22T03:48:34.044404+0000","last_undegraded":"2024-09-22T03:48:34.044404+0000","last_fullsized":"2024-09-22T03:48:34.044404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.040677+0000","last_change":"2024-09-22T03:48:34.046206+0000","last_active":"2024-09-22T03:48:34.040677+0000","last_peered":"2024-09-22T03:48:34.040677+0000","last_clean":"2024-09-22T03:48:34.040677+0000","last_became_active":"2024-09-22T03:48:32.229128+0000","last_became_peered":"2024-09-22T03:48:32.229128+0000","last_unstale":"2024-09-22T03:48:34.040677+0000","last_undegraded":"2024-09-22T03:48:34.040677+0000","last_fullsized":"2024-09-22T03:48:34.040677+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.074002+0000","last_change":"2024-09-22T03:48:34.083612+0000","last_active":"2024-09-22T03:48:34.074002+0000","last_peered":"2024-09-22T03:48:34.074002+0000","last_clean":"2024-09-22T03:48:34.074002+0000","last_became_active":"2024-09-22T03:48:32.242416+0000","last_became_peered":"2024-09-22T03:48:32.242416+0000","last_unstale":"2024-09-22T03:48:34.074002+0000","last_undegraded":"2024-09-22T03:48:34.074002+0000","last_fullsized":"2024-09-22T03:48:34.074002+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044429+0000","last_change":"2024-09-22T03:48:34.046827+0000","last_active":"2024-09-22T03:48:34.044429+0000","last_peered":"2024-09-22T03:48:34.044429+0000","last_clean":"2024-09-22T03:48:34.044429+0000","last_became_active":"2024-09-22T03:48:32.161038+0000","last_became_peered":"2024-09-22T03:48:32.161038+0000","last_unstale":"2024-09-22T03:48:34.044429+0000","last_undegraded":"2024-09-22T03:48:34.044429+0000","last_fullsized":"2024-09-22T03:48:34.044429+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.079953+0000","last_change":"2024-09-22T03:48:32.205697+0000","last_active":"2024-09-22T03:48:34.079953+0000","last_peered":"2024-09-22T03:48:34.079953+0000","last_clean":"2024-09-22T03:48:34.079953+0000","last_became_active":"2024-09-22T03:48:32.172058+0000","last_became_peered":"2024-09-22T03:48:32.172058+0000","last_unstale":"2024-09-22T03:48:34.079953+0000","last_undegraded":"2024-09-22T03:48:34.079953+0000","last_fullsized":"2024-09-22T03:48:34.079953+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044833+0000","last_change":"2024-09-22T03:48:34.051084+0000","last_active":"2024-09-22T03:48:34.044833+0000","last_peered":"2024-09-22T03:48:34.044833+0000","last_clean":"2024-09-22T03:48:34.044833+0000","last_became_active":"2024-09-22T03:48:32.223850+0000","last_became_peered":"2024-09-22T03:48:32.223850+0000","last_unstale":"2024-09-22T03:48:34.044833+0000","last_undegraded":"2024-09-22T03:48:34.044833+0000","last_fullsized":"2024-09-22T03:48:34.044833+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.056972+0000","last_change":"2024-09-22T03:48:34.056972+0000","last_active":"2024-09-22T03:48:34.056972+0000","last_peered":"2024-09-22T03:48:34.056972+0000","last_clean":"2024-09-22T03:48:34.056972+0000","last_became_active":"2024-09-22T03:48:32.122121+0000","last_became_peered":"2024-09-22T03:48:32.122121+0000","last_unstale":"2024-09-22T03:48:34.056972+0000","last_undegraded":"2024-09-22T03:48:34.056972+0000","last_fullsized":"2024-09-22T03:48:34.056972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.073680+0000","last_change":"2024-09-22T03:48:34.083728+0000","last_active":"2024-09-22T03:48:34.073680+0000","last_peered":"2024-09-22T03:48:34.073680+0000","last_clean":"2024-09-22T03:48:34.073680+0000","last_became_active":"2024-09-22T03:48:32.294179+0000","last_became_peered":"2024-09-22T03:48:32.294179+0000","last_unstale":"2024-09-22T03:48:34.073680+0000","last_undegraded":"2024-09-22T03:48:34.073680+0000","last_fullsized":"2024-09-22T03:48:34.073680+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.040871+0000","last_change":"2024-09-22T03:48:34.046112+0000","last_active":"2024-09-22T03:48:34.040871+0000","last_peered":"2024-09-22T03:48:34.040871+0000","last_clean":"2024-09-22T03:48:34.040871+0000","last_became_active":"2024-09-22T03:48:32.224335+0000","last_became_peered":"2024-09-22T03:48:32.224335+0000","last_unstale":"2024-09-22T03:48:34.040871+0000","last_undegraded":"2024-09-22T03:48:34.040871+0000","last_fullsized":"2024-09-22T03:48:34.040871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:48:41.761 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:48:41.868 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.863+0000 7fa56a0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:41.870 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:41.869+0000 7fa56a0d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:42.055 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:42.056 INFO:teuthology.orchestra.run.smithi083.stderr:dumped all 2024-09-22T03:48:42.068 INFO:teuthology.orchestra.run.smithi083.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T03:48:40.938960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.799687"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044404+0000","last_change":"2024-09-22T03:48:34.050664+0000","last_active":"2024-09-22T03:48:34.044404+0000","last_peered":"2024-09-22T03:48:34.044404+0000","last_clean":"2024-09-22T03:48:34.044404+0000","last_became_active":"2024-09-22T03:48:32.164891+0000","last_became_peered":"2024-09-22T03:48:32.164891+0000","last_unstale":"2024-09-22T03:48:34.044404+0000","last_undegraded":"2024-09-22T03:48:34.044404+0000","last_fullsized":"2024-09-22T03:48:34.044404+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.040677+0000","last_change":"2024-09-22T03:48:34.046206+0000","last_active":"2024-09-22T03:48:34.040677+0000","last_peered":"2024-09-22T03:48:34.040677+0000","last_clean":"2024-09-22T03:48:34.040677+0000","last_became_active":"2024-09-22T03:48:32.229128+0000","last_became_peered":"2024-09-22T03:48:32.229128+0000","last_unstale":"2024-09-22T03:48:34.040677+0000","last_undegraded":"2024-09-22T03:48:34.040677+0000","last_fullsized":"2024-09-22T03:48:34.040677+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.074002+0000","last_change":"2024-09-22T03:48:34.083612+0000","last_active":"2024-09-22T03:48:34.074002+0000","last_peered":"2024-09-22T03:48:34.074002+0000","last_clean":"2024-09-22T03:48:34.074002+0000","last_became_active":"2024-09-22T03:48:32.242416+0000","last_became_peered":"2024-09-22T03:48:32.242416+0000","last_unstale":"2024-09-22T03:48:34.074002+0000","last_undegraded":"2024-09-22T03:48:34.074002+0000","last_fullsized":"2024-09-22T03:48:34.074002+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044429+0000","last_change":"2024-09-22T03:48:34.046827+0000","last_active":"2024-09-22T03:48:34.044429+0000","last_peered":"2024-09-22T03:48:34.044429+0000","last_clean":"2024-09-22T03:48:34.044429+0000","last_became_active":"2024-09-22T03:48:32.161038+0000","last_became_peered":"2024-09-22T03:48:32.161038+0000","last_unstale":"2024-09-22T03:48:34.044429+0000","last_undegraded":"2024-09-22T03:48:34.044429+0000","last_fullsized":"2024-09-22T03:48:34.044429+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.079953+0000","last_change":"2024-09-22T03:48:32.205697+0000","last_active":"2024-09-22T03:48:34.079953+0000","last_peered":"2024-09-22T03:48:34.079953+0000","last_clean":"2024-09-22T03:48:34.079953+0000","last_became_active":"2024-09-22T03:48:32.172058+0000","last_became_peered":"2024-09-22T03:48:32.172058+0000","last_unstale":"2024-09-22T03:48:34.079953+0000","last_undegraded":"2024-09-22T03:48:34.079953+0000","last_fullsized":"2024-09-22T03:48:34.079953+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.044833+0000","last_change":"2024-09-22T03:48:34.051084+0000","last_active":"2024-09-22T03:48:34.044833+0000","last_peered":"2024-09-22T03:48:34.044833+0000","last_clean":"2024-09-22T03:48:34.044833+0000","last_became_active":"2024-09-22T03:48:32.223850+0000","last_became_peered":"2024-09-22T03:48:32.223850+0000","last_unstale":"2024-09-22T03:48:34.044833+0000","last_undegraded":"2024-09-22T03:48:34.044833+0000","last_fullsized":"2024-09-22T03:48:34.044833+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.056972+0000","last_change":"2024-09-22T03:48:34.056972+0000","last_active":"2024-09-22T03:48:34.056972+0000","last_peered":"2024-09-22T03:48:34.056972+0000","last_clean":"2024-09-22T03:48:34.056972+0000","last_became_active":"2024-09-22T03:48:32.122121+0000","last_became_peered":"2024-09-22T03:48:32.122121+0000","last_unstale":"2024-09-22T03:48:34.056972+0000","last_undegraded":"2024-09-22T03:48:34.056972+0000","last_fullsized":"2024-09-22T03:48:34.056972+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.073680+0000","last_change":"2024-09-22T03:48:34.083728+0000","last_active":"2024-09-22T03:48:34.073680+0000","last_peered":"2024-09-22T03:48:34.073680+0000","last_clean":"2024-09-22T03:48:34.073680+0000","last_became_active":"2024-09-22T03:48:32.294179+0000","last_became_peered":"2024-09-22T03:48:32.294179+0000","last_unstale":"2024-09-22T03:48:34.073680+0000","last_undegraded":"2024-09-22T03:48:34.073680+0000","last_fullsized":"2024-09-22T03:48:34.073680+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:48:34.040871+0000","last_change":"2024-09-22T03:48:34.046112+0000","last_active":"2024-09-22T03:48:34.040871+0000","last_peered":"2024-09-22T03:48:34.040871+0000","last_clean":"2024-09-22T03:48:34.040871+0000","last_became_active":"2024-09-22T03:48:32.224335+0000","last_became_peered":"2024-09-22T03:48:32.224335+0000","last_unstale":"2024-09-22T03:48:34.040871+0000","last_undegraded":"2024-09-22T03:48:34.040871+0000","last_fullsized":"2024-09-22T03:48:34.040871+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:48:31.008063+0000","last_clean_scrub_stamp":"2024-09-22T03:48:31.008063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:48:42.069 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:48:42.069 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:48:42.069 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:48:42.070 DEBUG:teuthology.orchestra.run.smithi083:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:48:42.176 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:42.172+0000 7fb806d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:42.178 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22T03:48:42.176+0000 7fb806d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:48:42.404 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:42.404 INFO:teuthology.orchestra.run.smithi083.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:48:42.417 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:48:42.417 INFO:teuthology.run_tasks:Running task cbt... 2024-09-22T03:48:42.433 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi083.front.sepia.ceph.com', 'osds': ['ubuntu@smithi083.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi083.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi083.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi083.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-22T03:48:42.433 DEBUG:teuthology.orchestra.run.smithi083:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-22T03:48:42.451 DEBUG:teuthology.orchestra.run.smithi083:> set -ex 2024-09-22T03:48:42.451 DEBUG:teuthology.orchestra.run.smithi083:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:48:42.503 DEBUG:teuthology.orchestra.run.smithi083:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-22T03:48:42.559 INFO:teuthology.orchestra.run.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-22T03:48:42.873 INFO:tasks.cbt:Installing collectl 2024-09-22T03:48:42.873 DEBUG:teuthology.orchestra.run.smithi083:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-22T03:48:42.973 INFO:teuthology.orchestra.run.smithi083.stderr:--2024-09-22 03:48:42-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-22T03:48:43.148 INFO:teuthology.orchestra.run.smithi083.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-09-22T03:48:43.156 INFO:teuthology.orchestra.run.smithi083.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-09-22T03:48:43.386 INFO:teuthology.orchestra.run.smithi083.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:48:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75ObmpBzSb6YIVSVxOWWWUx5w13eRL37MSTeCbxtVkLhzIGzFaj2z93dPHZRGZBpkMZ2HRr7g6INu0Nke627F-EPPA%3D%3D&use_mirror=pilotfiber&r= [following] 2024-09-22T03:48:43.387 INFO:teuthology.orchestra.run.smithi083.stderr:--2024-09-22 03:48:43-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75ObmpBzSb6YIVSVxOWWWUx5w13eRL37MSTeCbxtVkLhzIGzFaj2z93dPHZRGZBpkMZ2HRr7g6INu0Nke627F-EPPA%3D%3D&use_mirror=pilotfiber&r= 2024-09-22T03:48:43.396 INFO:teuthology.orchestra.run.smithi083.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-22T03:48:43.474 INFO:teuthology.orchestra.run.smithi083.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-22T03:48:43.800 INFO:teuthology.orchestra.run.smithi083.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:48:43.800 INFO:teuthology.orchestra.run.smithi083.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-22T03:48:43.800 INFO:teuthology.orchestra.run.smithi083.stderr:--2024-09-22 03:48:43-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-22T03:48:43.810 INFO:teuthology.orchestra.run.smithi083.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2024-09-22T03:48:43.823 INFO:teuthology.orchestra.run.smithi083.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2024-09-22T03:48:43.932 INFO:teuthology.orchestra.run.smithi083.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-22T03:48:43.932 INFO:teuthology.orchestra.run.smithi083.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-22T03:48:43.932 INFO:teuthology.orchestra.run.smithi083.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-22T03:48:43.932 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:43.945 INFO:teuthology.orchestra.run.smithi083.stderr: 0K .......... .......... .......... .......... .......... 8% 3.76M 0s 2024-09-22T03:48:43.971 INFO:teuthology.orchestra.run.smithi083.stderr: 50K .......... .......... .......... .......... .......... 16% 1.91M 0s 2024-09-22T03:48:43.971 INFO:teuthology.orchestra.run.smithi083.stderr: 100K .......... .......... .......... .......... .......... 24% 218M 0s 2024-09-22T03:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr: 150K .......... .......... .......... .......... .......... 32% 3.80M 0s 2024-09-22T03:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr: 200K .......... .......... .......... .......... .......... 41% 300M 0s 2024-09-22T03:48:43.986 INFO:teuthology.orchestra.run.smithi083.stderr: 250K .......... .......... .......... .......... .......... 49% 390M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 300K .......... .......... .......... .......... .......... 57% 3.97M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 350K .......... .......... .......... .......... .......... 65% 232M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 400K .......... .......... .......... .......... .......... 74% 312M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 450K .......... .......... .......... .......... .......... 82% 386M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 500K .......... .......... .......... .......... .......... 90% 369M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 550K .......... .......... .......... .......... .......... 98% 345M 0s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 600K ...... 100% 11.4T=0.06s 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:44.001 INFO:teuthology.orchestra.run.smithi083.stderr:2024-09-22 03:48:43 (9.11 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-22T03:48:44.002 INFO:teuthology.orchestra.run.smithi083.stderr: 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/ 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/collectl.conf 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/ 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-22T03:48:44.034 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Socket.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/colmux.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Import.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/style.css 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Process.html 2024-09-22T03:48:44.035 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Export.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Startup.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Performance.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Messages.html 2024-09-22T03:48:44.036 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Logging.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Misc.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Examples.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Data.html 2024-09-22T03:48:44.037 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Playback.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Hello.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Network.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Memory.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/Disks.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/service/ 2024-09-22T03:48:44.038 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/service/collectl.service 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/initd/ 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/initd/collectl 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/misc.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/gexpr.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/statsd.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/envrules.std 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/hello.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/vmsum.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/proctree.ph 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/README-WINDOWS 2024-09-22T03:48:44.039 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/formatit.ph 2024-09-22T03:48:44.040 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/colmux 2024-09-22T03:48:44.040 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/INSTALL 2024-09-22T03:48:44.040 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/graphite.ph 2024-09-22T03:48:44.040 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-22T03:48:44.041 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/vmstat.ph 2024-09-22T03:48:44.041 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/README 2024-09-22T03:48:44.041 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/man1/ 2024-09-22T03:48:44.041 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/man1/collectl.1 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/man1/colmux.1 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/lexpr.ph 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/client.pl 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/collectl 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/COPYING 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/vnet.ph 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/GPL 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/UNINSTALL 2024-09-22T03:48:44.042 INFO:teuthology.orchestra.run.smithi083.stdout:collectl-4.3.1/ARTISTIC 2024-09-22T03:48:44.090 INFO:teuthology.orchestra.run.smithi083.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-22T03:48:44.092 DEBUG:teuthology.orchestra.run.smithi083:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-22T03:48:44.694 INFO:teuthology.orchestra.run.smithi083.stdout:Last metadata expiration check: 0:03:05 ago on Sun 22 Sep 2024 03:45:39 AM UTC. 2024-09-22T03:48:44.787 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-22T03:48:44.789 INFO:teuthology.orchestra.run.smithi083.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-22T03:48:44.838 INFO:teuthology.orchestra.run.smithi083.stdout:Dependencies resolved. 2024-09-22T03:48:44.838 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:44.838 INFO:teuthology.orchestra.run.smithi083.stdout: Package Arch Version Repository Size 2024-09-22T03:48:44.838 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout:Installing: 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout:Installing dependencies: 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction Summary 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout:================================================================================ 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout:Install 5 Packages 2024-09-22T03:48:44.839 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:44.840 INFO:teuthology.orchestra.run.smithi083.stdout:Total download size: 2.4 M 2024-09-22T03:48:44.840 INFO:teuthology.orchestra.run.smithi083.stdout:Installed size: 11 M 2024-09-22T03:48:44.840 INFO:teuthology.orchestra.run.smithi083.stdout:Downloading Packages: 2024-09-22T03:48:45.194 INFO:teuthology.orchestra.run.smithi083.stdout:(1/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 131 kB/s | 24 kB 00:00 2024-09-22T03:48:45.219 INFO:teuthology.orchestra.run.smithi083.stdout:(2/5): libradospp-devel-19.3.0-5092.g62da54b4.e 148 kB/s | 31 kB 00:00 2024-09-22T03:48:45.261 INFO:teuthology.orchestra.run.smithi083.stdout:(3/5): perf-5.14.0-508.el9.x86_64.rpm 9.0 MB/s | 2.2 MB 00:00 2024-09-22T03:48:45.311 INFO:teuthology.orchestra.run.smithi083.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 1.0 MB/s | 118 kB 00:00 2024-09-22T03:48:45.336 INFO:teuthology.orchestra.run.smithi083.stdout:(5/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 112 kB/s | 13 kB 00:00 2024-09-22T03:48:45.339 INFO:teuthology.orchestra.run.smithi083.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:48:45.339 INFO:teuthology.orchestra.run.smithi083.stdout:Total 4.8 MB/s | 2.4 MB 00:00 2024-09-22T03:48:45.369 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction check 2024-09-22T03:48:45.390 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction check succeeded. 2024-09-22T03:48:45.390 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction test 2024-09-22T03:48:45.519 INFO:teuthology.orchestra.run.smithi083.stdout:Transaction test succeeded. 2024-09-22T03:48:45.520 INFO:teuthology.orchestra.run.smithi083.stdout:Running transaction 2024-09-22T03:48:45.865 INFO:teuthology.orchestra.run.smithi083.stdout: Preparing : 1/1 2024-09-22T03:48:45.938 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-22T03:48:45.969 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-22T03:48:46.000 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-22T03:48:46.336 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-22T03:48:46.406 INFO:teuthology.orchestra.run.smithi083.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:48:48.211 INFO:teuthology.orchestra.run.smithi083.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:48:48.211 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-22T03:48:48.211 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-22T03:48:48.211 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-22T03:48:48.211 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout:Installed: 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: pdsh-2.34-7.el9.x86_64 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: perf-5.14.0-508.el9.x86_64 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:48.562 INFO:teuthology.orchestra.run.smithi083.stdout:Complete! 2024-09-22T03:48:48.932 INFO:tasks.cbt:benchmark: librbdfio 2024-09-22T03:48:48.932 DEBUG:teuthology.orchestra.run.smithi083:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-09-22T03:48:48.947 INFO:teuthology.orchestra.run.smithi083.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-09-22T03:48:51.145 DEBUG:teuthology.orchestra.run.smithi083:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:Operating system Linux 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:CPU x86_64 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:Big endian no 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:Compiler gcc 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:Cross compile no 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout: 2024-09-22T03:48:51.717 INFO:teuthology.orchestra.run.smithi083.stdout:Static build no 2024-09-22T03:48:51.826 INFO:teuthology.orchestra.run.smithi083.stdout:Wordsize 64 2024-09-22T03:48:51.833 INFO:teuthology.orchestra.run.smithi083.stdout:zlib no 2024-09-22T03:48:51.873 INFO:teuthology.orchestra.run.smithi083.stdout:fcntl(F_FULLFSYNC) no 2024-09-22T03:48:51.887 INFO:teuthology.orchestra.run.smithi083.stdout:Linux AIO support no 2024-09-22T03:48:51.887 INFO:teuthology.orchestra.run.smithi083.stdout:Linux AIO support rw flags no 2024-09-22T03:48:51.942 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX AIO support yes 2024-09-22T03:48:51.942 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX AIO support needs -lrt no 2024-09-22T03:48:51.973 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX AIO fsync yes 2024-09-22T03:48:52.003 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX pshared support yes 2024-09-22T03:48:52.056 INFO:teuthology.orchestra.run.smithi083.stdout:pthread_condattr_setclock() yes 2024-09-22T03:48:52.104 INFO:teuthology.orchestra.run.smithi083.stdout:pthread_sigmask() yes 2024-09-22T03:48:52.140 INFO:teuthology.orchestra.run.smithi083.stdout:pthread_getaffinity_np() yes 2024-09-22T03:48:52.151 INFO:teuthology.orchestra.run.smithi083.stdout:Solaris AIO support no 2024-09-22T03:48:52.201 INFO:teuthology.orchestra.run.smithi083.stdout:__sync_fetch_and_add yes 2024-09-22T03:48:52.239 INFO:teuthology.orchestra.run.smithi083.stdout:__sync_synchronize yes 2024-09-22T03:48:52.268 INFO:teuthology.orchestra.run.smithi083.stdout:__sync_val_compare_and_swap yes 2024-09-22T03:48:52.275 INFO:teuthology.orchestra.run.smithi083.stdout:libverbs no 2024-09-22T03:48:52.308 INFO:teuthology.orchestra.run.smithi083.stdout:rdmacm no 2024-09-22T03:48:52.339 INFO:teuthology.orchestra.run.smithi083.stdout:asprintf() yes 2024-09-22T03:48:52.369 INFO:teuthology.orchestra.run.smithi083.stdout:vasprintf() yes 2024-09-22T03:48:52.400 INFO:teuthology.orchestra.run.smithi083.stdout:Linux fallocate yes 2024-09-22T03:48:52.431 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX fadvise yes 2024-09-22T03:48:52.463 INFO:teuthology.orchestra.run.smithi083.stdout:POSIX fallocate yes 2024-09-22T03:48:52.492 INFO:teuthology.orchestra.run.smithi083.stdout:sched_setaffinity(3 arg) yes 2024-09-22T03:48:52.493 INFO:teuthology.orchestra.run.smithi083.stdout:sched_setaffinity(2 arg) no 2024-09-22T03:48:52.523 INFO:teuthology.orchestra.run.smithi083.stdout:clock_gettime yes 2024-09-22T03:48:52.554 INFO:teuthology.orchestra.run.smithi083.stdout:CLOCK_MONOTONIC yes 2024-09-22T03:48:52.586 INFO:teuthology.orchestra.run.smithi083.stdout:clockid_t yes 2024-09-22T03:48:52.632 INFO:teuthology.orchestra.run.smithi083.stdout:gettimeofday yes 2024-09-22T03:48:52.665 INFO:teuthology.orchestra.run.smithi083.stdout:fdatasync yes 2024-09-22T03:48:52.696 INFO:teuthology.orchestra.run.smithi083.stdout:pipe() yes 2024-09-22T03:48:52.726 INFO:teuthology.orchestra.run.smithi083.stdout:pipe2() yes 2024-09-22T03:48:52.757 INFO:teuthology.orchestra.run.smithi083.stdout:pread() yes 2024-09-22T03:48:52.839 INFO:teuthology.orchestra.run.smithi083.stdout:sync_file_range yes 2024-09-22T03:48:52.847 INFO:teuthology.orchestra.run.smithi083.stdout:ASharedMemory_create no 2024-09-22T03:48:52.869 INFO:teuthology.orchestra.run.smithi083.stdout:EXT4 move extent yes 2024-09-22T03:48:52.900 INFO:teuthology.orchestra.run.smithi083.stdout:Linux splice(2) yes 2024-09-22T03:48:52.908 INFO:teuthology.orchestra.run.smithi083.stdout:libnuma no 2024-09-22T03:48:52.937 INFO:teuthology.orchestra.run.smithi083.stdout:strsep yes 2024-09-22T03:48:52.967 INFO:teuthology.orchestra.run.smithi083.stdout:strcasestr yes 2024-09-22T03:48:52.982 INFO:teuthology.orchestra.run.smithi083.stdout:strlcat no 2024-09-22T03:48:53.015 INFO:teuthology.orchestra.run.smithi083.stdout:getopt_long_only() yes 2024-09-22T03:48:53.093 INFO:teuthology.orchestra.run.smithi083.stdout:inet_aton yes 2024-09-22T03:48:53.125 INFO:teuthology.orchestra.run.smithi083.stdout:socklen_t yes 2024-09-22T03:48:53.156 INFO:teuthology.orchestra.run.smithi083.stdout:__thread yes 2024-09-22T03:48:53.199 INFO:teuthology.orchestra.run.smithi083.stdout:RUSAGE_THREAD yes 2024-09-22T03:48:53.228 INFO:teuthology.orchestra.run.smithi083.stdout:SCHED_IDLE yes 2024-09-22T03:48:53.262 INFO:teuthology.orchestra.run.smithi083.stdout:TCP_NODELAY yes 2024-09-22T03:48:53.322 INFO:teuthology.orchestra.run.smithi083.stdout:vsock yes 2024-09-22T03:48:53.356 INFO:teuthology.orchestra.run.smithi083.stdout:Net engine window_size yes 2024-09-22T03:48:53.391 INFO:teuthology.orchestra.run.smithi083.stdout:TCP_MAXSEG yes 2024-09-22T03:48:53.421 INFO:teuthology.orchestra.run.smithi083.stdout:RLIMIT_MEMLOCK yes 2024-09-22T03:48:53.454 INFO:teuthology.orchestra.run.smithi083.stdout:pwritev/preadv yes 2024-09-22T03:48:53.486 INFO:teuthology.orchestra.run.smithi083.stdout:pwritev2/preadv2 yes 2024-09-22T03:48:53.523 INFO:teuthology.orchestra.run.smithi083.stdout:IPv6 helpers yes 2024-09-22T03:48:53.539 INFO:teuthology.orchestra.run.smithi083.stdout:http engine no 2024-09-22T03:48:53.682 INFO:teuthology.orchestra.run.smithi083.stdout:Rados engine yes 2024-09-22T03:48:53.881 INFO:teuthology.orchestra.run.smithi083.stdout:Rados Block Device engine yes 2024-09-22T03:48:54.063 INFO:teuthology.orchestra.run.smithi083.stdout:rbd_poll yes 2024-09-22T03:48:54.250 INFO:teuthology.orchestra.run.smithi083.stdout:rbd_invalidate_cache yes 2024-09-22T03:48:54.280 INFO:teuthology.orchestra.run.smithi083.stdout:setvbuf yes 2024-09-22T03:48:54.287 INFO:teuthology.orchestra.run.smithi083.stdout:Gluster API engine no 2024-09-22T03:48:54.303 INFO:teuthology.orchestra.run.smithi083.stdout:s390_z196_facilities no 2024-09-22T03:48:54.304 INFO:teuthology.orchestra.run.smithi083.stdout:HDFS engine no 2024-09-22T03:48:54.363 INFO:teuthology.orchestra.run.smithi083.stdout:MTD yes 2024-09-22T03:48:54.371 INFO:teuthology.orchestra.run.smithi083.stdout:libpmem no 2024-09-22T03:48:54.371 INFO:teuthology.orchestra.run.smithi083.stdout:libpmem1_5 no 2024-09-22T03:48:54.379 INFO:teuthology.orchestra.run.smithi083.stdout:libpmem2 no 2024-09-22T03:48:54.379 INFO:teuthology.orchestra.run.smithi083.stdout:PMDK dev-dax engine no 2024-09-22T03:48:54.379 INFO:teuthology.orchestra.run.smithi083.stdout:PMDK libpmem engine no 2024-09-22T03:48:54.387 INFO:teuthology.orchestra.run.smithi083.stdout:DDN's Infinite Memory Engine no 2024-09-22T03:48:54.387 INFO:teuthology.orchestra.run.smithi083.stdout:iscsi engine no 2024-09-22T03:48:54.387 INFO:teuthology.orchestra.run.smithi083.stdout:NBD engine no 2024-09-22T03:48:54.396 INFO:teuthology.orchestra.run.smithi083.stdout:DAOS File System (dfs) Engine no 2024-09-22T03:48:54.407 INFO:teuthology.orchestra.run.smithi083.stdout:NFS engine no 2024-09-22T03:48:54.407 INFO:teuthology.orchestra.run.smithi083.stdout:lex/yacc for arithmetic no 2024-09-22T03:48:54.438 INFO:teuthology.orchestra.run.smithi083.stdout:getmntent yes 2024-09-22T03:48:54.473 INFO:teuthology.orchestra.run.smithi083.stdout:getmntinfo no 2024-09-22T03:48:54.529 INFO:teuthology.orchestra.run.smithi083.stdout:Static Assert yes 2024-09-22T03:48:54.557 INFO:teuthology.orchestra.run.smithi083.stdout:bool yes 2024-09-22T03:48:54.591 INFO:teuthology.orchestra.run.smithi083.stdout:strndup yes 2024-09-22T03:48:54.598 INFO:teuthology.orchestra.run.smithi083.stdout:Valgrind headers no 2024-09-22T03:48:54.627 INFO:teuthology.orchestra.run.smithi083.stdout:Zoned block device support yes 2024-09-22T03:48:54.655 INFO:teuthology.orchestra.run.smithi083.stdout:Zoned block device capacity yes 2024-09-22T03:48:54.664 INFO:teuthology.orchestra.run.smithi083.stdout:libzbc engine no 2024-09-22T03:48:54.702 INFO:teuthology.orchestra.run.smithi083.stdout:NVMe uring command support yes 2024-09-22T03:48:54.717 INFO:teuthology.orchestra.run.smithi083.stdout:xnvme engine no 2024-09-22T03:48:54.725 INFO:teuthology.orchestra.run.smithi083.stdout:isal no 2024-09-22T03:48:54.740 INFO:teuthology.orchestra.run.smithi083.stdout:libblkio engine no 2024-09-22T03:48:54.740 INFO:teuthology.orchestra.run.smithi083.stdout:march_armv8_a_crc_crypto no 2024-09-22T03:48:54.740 INFO:teuthology.orchestra.run.smithi083.stdout:cuda no 2024-09-22T03:48:54.740 INFO:teuthology.orchestra.run.smithi083.stdout:libcufile no 2024-09-22T03:48:54.767 INFO:teuthology.orchestra.run.smithi083.stdout:Build march=native yes 2024-09-22T03:48:54.775 INFO:teuthology.orchestra.run.smithi083.stdout:CUnit no 2024-09-22T03:48:54.803 INFO:teuthology.orchestra.run.smithi083.stdout:__kernel_rwf_t yes 2024-09-22T03:48:54.830 INFO:teuthology.orchestra.run.smithi083.stdout:-Wimplicit-fallthrough=2 yes 2024-09-22T03:48:54.861 INFO:teuthology.orchestra.run.smithi083.stdout:-Wno-stringop-truncation yes 2024-09-22T03:48:54.900 INFO:teuthology.orchestra.run.smithi083.stdout:MADV_HUGEPAGE yes 2024-09-22T03:48:54.935 INFO:teuthology.orchestra.run.smithi083.stdout:gettid yes 2024-09-22T03:48:54.966 INFO:teuthology.orchestra.run.smithi083.stdout:statx(2)/libc yes 2024-09-22T03:48:55.001 INFO:teuthology.orchestra.run.smithi083.stdout:statx(2)/syscall yes 2024-09-22T03:48:55.003 INFO:teuthology.orchestra.run.smithi083.stdout:Windows PDB generation no 2024-09-22T03:48:55.035 INFO:teuthology.orchestra.run.smithi083.stdout:timerfd_create yes 2024-09-22T03:48:55.036 INFO:teuthology.orchestra.run.smithi083.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-09-22T03:48:55.037 INFO:teuthology.orchestra.run.smithi083.stdout:Lib-based ioengines dynamic no 2024-09-22T03:48:55.100 INFO:teuthology.orchestra.run.smithi083.stdout:TCMalloc support yes 2024-09-22T03:48:55.101 INFO:teuthology.orchestra.run.smithi083.stdout:seed_buckets 4 2024-09-22T03:48:55.178 INFO:teuthology.orchestra.run.smithi083.stderr:FIO_VERSION = fio-3.37-124-gbcd4 2024-09-22T03:48:55.213 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc16.o 2024-09-22T03:48:55.310 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc32.o 2024-09-22T03:48:55.346 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc32c-arm64.o 2024-09-22T03:48:55.432 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc32c-intel.o 2024-09-22T03:48:55.478 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc32c.o 2024-09-22T03:48:55.517 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc64.o 2024-09-22T03:48:55.557 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crc7.o 2024-09-22T03:48:55.590 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/crct10dif_common.o 2024-09-22T03:48:55.626 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/fnv.o 2024-09-22T03:48:55.666 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/md5.o 2024-09-22T03:48:55.768 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/murmur3.o 2024-09-22T03:48:55.813 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/sha1.o 2024-09-22T03:48:56.002 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/sha256.o 2024-09-22T03:48:56.308 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/sha3.o 2024-09-22T03:48:56.467 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/sha512.o 2024-09-22T03:48:56.583 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/test.o 2024-09-22T03:48:56.744 INFO:teuthology.orchestra.run.smithi083.stdout: CC crc/xxhash.o 2024-09-22T03:48:56.842 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/axmap.o 2024-09-22T03:48:56.965 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/bloom.o 2024-09-22T03:48:57.050 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/flist_sort.o 2024-09-22T03:48:57.122 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/gauss.o 2024-09-22T03:48:57.213 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/getrusage.o 2024-09-22T03:48:57.249 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/hweight.o 2024-09-22T03:48:57.288 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/ieee754.o 2024-09-22T03:48:57.337 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/lfsr.o 2024-09-22T03:48:57.439 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/memalign.o 2024-09-22T03:48:57.483 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/memcpy.o 2024-09-22T03:48:57.629 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/mountcheck.o 2024-09-22T03:48:57.672 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/num2str.o 2024-09-22T03:48:57.739 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/output_buffer.o 2024-09-22T03:48:57.787 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/pattern.o 2024-09-22T03:48:57.930 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/prio_tree.o 2024-09-22T03:48:58.052 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/rand.o 2024-09-22T03:48:58.244 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/rbtree.o 2024-09-22T03:48:58.370 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/strntol.o 2024-09-22T03:48:58.418 INFO:teuthology.orchestra.run.smithi083.stdout: CC lib/zipf.o 2024-09-22T03:48:58.537 INFO:teuthology.orchestra.run.smithi083.stdout: CC gettime.o 2024-09-22T03:48:58.828 INFO:teuthology.orchestra.run.smithi083.stdout: CC ioengines.o 2024-09-22T03:48:59.095 INFO:teuthology.orchestra.run.smithi083.stdout: CC init.o 2024-09-22T03:48:59.864 INFO:teuthology.orchestra.run.smithi083.stdout: CC stat.o 2024-09-22T03:49:01.694 INFO:teuthology.orchestra.run.smithi083.stdout: CC log.o 2024-09-22T03:49:01.826 INFO:teuthology.orchestra.run.smithi083.stdout: CC time.o 2024-09-22T03:49:01.953 INFO:teuthology.orchestra.run.smithi083.stdout: CC filesetup.o 2024-09-22T03:49:02.648 INFO:teuthology.orchestra.run.smithi083.stdout: CC eta.o 2024-09-22T03:49:02.968 INFO:teuthology.orchestra.run.smithi083.stdout: CC verify.o 2024-09-22T03:49:03.654 INFO:teuthology.orchestra.run.smithi083.stdout: CC memory.o 2024-09-22T03:49:03.822 INFO:teuthology.orchestra.run.smithi083.stdout: CC io_u.o 2024-09-22T03:49:04.619 INFO:teuthology.orchestra.run.smithi083.stdout: CC parse.o 2024-09-22T03:49:05.065 INFO:teuthology.orchestra.run.smithi083.stdout: CC fio_sem.o 2024-09-22T03:49:05.171 INFO:teuthology.orchestra.run.smithi083.stdout: CC rwlock.o 2024-09-22T03:49:05.255 INFO:teuthology.orchestra.run.smithi083.stdout: CC pshared.o 2024-09-22T03:49:05.319 INFO:teuthology.orchestra.run.smithi083.stdout: CC options.o 2024-09-22T03:49:06.126 INFO:teuthology.orchestra.run.smithi083.stdout: CC smalloc.o 2024-09-22T03:49:06.312 INFO:teuthology.orchestra.run.smithi083.stdout: CC filehash.o 2024-09-22T03:49:06.463 INFO:teuthology.orchestra.run.smithi083.stdout: CC profile.o 2024-09-22T03:49:06.581 INFO:teuthology.orchestra.run.smithi083.stdout: CC debug.o 2024-09-22T03:49:06.623 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/cpu.o 2024-09-22T03:49:06.763 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/mmap.o 2024-09-22T03:49:06.927 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/sync.o 2024-09-22T03:49:07.159 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/null.o 2024-09-22T03:49:07.276 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/net.o 2024-09-22T03:49:07.709 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/ftruncate.o 2024-09-22T03:49:07.806 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/fileoperations.o 2024-09-22T03:49:07.962 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/exec.o 2024-09-22T03:49:08.114 INFO:teuthology.orchestra.run.smithi083.stdout: CC server.o 2024-09-22T03:49:08.861 INFO:teuthology.orchestra.run.smithi083.stdout: CC client.o 2024-09-22T03:49:09.398 INFO:teuthology.orchestra.run.smithi083.stdout: CC iolog.o 2024-09-22T03:49:09.848 INFO:teuthology.orchestra.run.smithi083.stdout: CC backend.o 2024-09-22T03:49:10.694 INFO:teuthology.orchestra.run.smithi083.stdout: CC libfio.o 2024-09-22T03:49:10.867 INFO:teuthology.orchestra.run.smithi083.stdout: CC flow.o 2024-09-22T03:49:10.982 INFO:teuthology.orchestra.run.smithi083.stdout: CC cconv.o 2024-09-22T03:49:11.337 INFO:teuthology.orchestra.run.smithi083.stdout: CC gettime-thread.o 2024-09-22T03:49:11.439 INFO:teuthology.orchestra.run.smithi083.stdout: CC helpers.o 2024-09-22T03:49:11.501 INFO:teuthology.orchestra.run.smithi083.stdout: CC json.o 2024-09-22T03:49:11.688 INFO:teuthology.orchestra.run.smithi083.stdout: CC idletime.o 2024-09-22T03:49:11.909 INFO:teuthology.orchestra.run.smithi083.stdout: CC td_error.o 2024-09-22T03:49:12.010 INFO:teuthology.orchestra.run.smithi083.stdout: CC profiles/tiobench.o 2024-09-22T03:49:12.113 INFO:teuthology.orchestra.run.smithi083.stdout: CC profiles/act.o 2024-09-22T03:49:12.280 INFO:teuthology.orchestra.run.smithi083.stdout: CC io_u_queue.o 2024-09-22T03:49:12.333 INFO:teuthology.orchestra.run.smithi083.stdout: CC filelock.o 2024-09-22T03:49:12.463 INFO:teuthology.orchestra.run.smithi083.stdout: CC workqueue.o 2024-09-22T03:49:12.641 INFO:teuthology.orchestra.run.smithi083.stdout: CC rate-submit.o 2024-09-22T03:49:12.791 INFO:teuthology.orchestra.run.smithi083.stdout: CC optgroup.o 2024-09-22T03:49:12.837 INFO:teuthology.orchestra.run.smithi083.stdout: CC helper_thread.o 2024-09-22T03:49:13.016 INFO:teuthology.orchestra.run.smithi083.stdout: CC steadystate.o 2024-09-22T03:49:13.250 INFO:teuthology.orchestra.run.smithi083.stdout: CC zone-dist.o 2024-09-22T03:49:13.382 INFO:teuthology.orchestra.run.smithi083.stdout: CC zbd.o 2024-09-22T03:49:14.071 INFO:teuthology.orchestra.run.smithi083.stdout: CC dedupe.o 2024-09-22T03:49:14.219 INFO:teuthology.orchestra.run.smithi083.stdout: CC dataplacement.o 2024-09-22T03:49:14.389 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/posixaio.o 2024-09-22T03:49:14.522 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/falloc.o 2024-09-22T03:49:14.635 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/e4defrag.o 2024-09-22T03:49:14.767 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/splice.o 2024-09-22T03:49:14.953 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/asprintf.o 2024-09-22T03:49:14.989 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/strlcat.o 2024-09-22T03:49:15.034 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/mtd.o 2024-09-22T03:49:15.199 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/libmtd.o 2024-09-22T03:49:15.582 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/libmtd_legacy.o 2024-09-22T03:49:15.714 INFO:teuthology.orchestra.run.smithi083.stdout: CC diskutil.o 2024-09-22T03:49:15.925 INFO:teuthology.orchestra.run.smithi083.stdout: CC fifo.o 2024-09-22T03:49:15.981 INFO:teuthology.orchestra.run.smithi083.stdout: CC blktrace.o 2024-09-22T03:49:16.319 INFO:teuthology.orchestra.run.smithi083.stdout: CC cgroup.o 2024-09-22T03:49:16.467 INFO:teuthology.orchestra.run.smithi083.stdout: CC trim.o 2024-09-22T03:49:16.582 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/sg.o 2024-09-22T03:49:16.961 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/linux-dev-lookup.o 2024-09-22T03:49:17.018 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/io_uring.o 2024-09-22T03:49:17.417 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/nvme.o 2024-09-22T03:49:17.725 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/linux-blkzoned.o 2024-09-22T03:49:17.880 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/cmdprio.o 2024-09-22T03:49:18.122 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/rados.o 2024-09-22T03:49:18.295 INFO:teuthology.orchestra.run.smithi083.stdout: CC engines/rbd.o 2024-09-22T03:49:18.475 INFO:teuthology.orchestra.run.smithi083.stdout: CC fio.o 2024-09-22T03:49:18.566 INFO:teuthology.orchestra.run.smithi083.stdout: LINK fio 2024-09-22T03:49:18.838 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/log.o 2024-09-22T03:49:18.880 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/genzipf.o 2024-09-22T03:49:19.022 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/strcasestr.o 2024-09-22T03:49:19.048 INFO:teuthology.orchestra.run.smithi083.stdout: CC oslib/strndup.o 2024-09-22T03:49:19.072 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/fio-genzipf 2024-09-22T03:49:19.248 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/btrace2fio.o 2024-09-22T03:49:19.663 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/fio-btrace2fio 2024-09-22T03:49:19.842 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/verify-state.o 2024-09-22T03:49:19.940 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/debug.o 2024-09-22T03:49:19.977 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/fio-verify-state 2024-09-22T03:49:20.152 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/stest.o 2024-09-22T03:49:20.236 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/arch.o 2024-09-22T03:49:20.267 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/stest 2024-09-22T03:49:20.440 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/ieee754.o 2024-09-22T03:49:20.480 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/ieee754 2024-09-22T03:49:20.649 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/axmap.o 2024-09-22T03:49:20.749 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/axmap 2024-09-22T03:49:20.919 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/lfsr-test.o 2024-09-22T03:49:21.017 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/lfsr-test 2024-09-22T03:49:21.193 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/gen-rand.o 2024-09-22T03:49:21.291 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/gen-rand 2024-09-22T03:49:21.467 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/memlock.o 2024-09-22T03:49:21.529 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/memlock 2024-09-22T03:49:21.701 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/read-to-pipe-async.o 2024-09-22T03:49:21.889 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/read-to-pipe-async 2024-09-22T03:49:22.060 INFO:teuthology.orchestra.run.smithi083.stdout: CC t/io_uring.o 2024-09-22T03:49:22.593 INFO:teuthology.orchestra.run.smithi083.stdout: LINK t/io_uring 2024-09-22T03:49:22.765 DEBUG:teuthology.orchestra.run.smithi083:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-22T03:49:22.791 DEBUG:teuthology.orchestra.run.smithi083:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:49:23.045 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - Settings.cluster: 2024-09-22T03:49:23.045 INFO:teuthology.orchestra.run.smithi083.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'clients': ['ubuntu@smithi083.front.sepia.ceph.com'], 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'head': 'ubuntu@smithi083.front.sepia.ceph.com', 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'is_teuthology': True, 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'iterations': 1, 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'mons': ['ubuntu@smithi083.front.sepia.ceph.com'], 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'osds': ['ubuntu@smithi083.front.sepia.ceph.com'], 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'osds_per_node': 3, 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'pid_dir': '/var/run/ceph', 2024-09-22T03:49:23.046 INFO:teuthology.orchestra.run.smithi083.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr: 'rebuild_every_test': False, 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr: 'rgws': {'ubuntu@smithi083.front.sepia.ceph.com': None}, 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr: 'use_existing': True, 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr: 'user': 'ubuntu'} 2024-09-22T03:49:23.047 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 2024-09-22T03:49:23.048 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2024-09-22T03:49:23.049 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-09-22T03:49:23.052 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-22T03:49:23.052 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-09-22T03:49:23.055 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:49:23.056 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-09-22T03:49:23.059 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:49:23.059 INFO:teuthology.orchestra.run.smithi083.stderr:03:49:23 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:50:23.112 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:50:23.458 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:50:23.502 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2024-09-22T03:50:23.517 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi083.front.sepia.ceph.com ; done 2024-09-22T03:50:23.522 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - INFO - cbt - Creating fio images... 2024-09-22T03:50:23.522 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-09-22T03:50:23.525 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:50:23.526 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-09-22T03:50:23.529 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:50:23.529 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:50:23.530 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-09-22T03:50:23.839 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-09-22T03:50:25.134 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-09-22T03:50:26.132 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:26 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-09-22T03:50:27.139 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:27 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-09-22T03:50:28.142 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:28 - INFO - cbt - Checking Health after pool creation. 2024-09-22T03:50:33.147 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:50:33.620 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:33 - INFO - cbt - Final Pool Health Check. 2024-09-22T03:50:38.624 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:50:38.949 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-09-22T03:50:38.952 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi083.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-09-22T03:50:40.534 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi083.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-09-22T03:50:40.715 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:50:40.740 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:50:40.775 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - INFO - cbt - Attempting to prefill fio images... 2024-09-22T03:50:40.776 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-22T03:50:40.777 INFO:teuthology.orchestra.run.smithi083.stderr:03:50:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-22T03:52:31.129 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:52:36.532 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-09-22T03:52:41.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-09-22T03:52:47.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-09-22T03:52:52.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-09-22T03:52:58.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-09-22T03:53:03.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-09-22T03:53:08.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2024-09-22T03:53:14.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2024-09-22T03:53:19.756 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-09-22T03:53:25.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2024-09-22T03:53:25.391 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:25.390+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:30.391 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:30.390+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:30.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2024-09-22T03:53:35.391 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:35.390+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:35.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-09-22T03:53:40.392 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:40.390+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:41.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-09-22T03:53:45.392 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:45.390+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:46.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2024-09-22T03:53:50.393 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:50.391+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:52.173 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2024-09-22T03:53:55.393 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:53:55.391+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:53:57.680 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~87s 2024-09-22T03:54:00.393 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:00.391+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:03.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-09-22T03:54:05.393 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:05.393+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:08.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2024-09-22T03:54:10.394 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:10.393+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:13.889 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~103s 2024-09-22T03:54:15.394 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:15.394+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:19.291 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~108s 2024-09-22T03:54:20.395 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:20.393+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:24.694 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~114s 2024-09-22T03:54:25.395 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:25.393+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:30.097 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~119s 2024-09-22T03:54:30.395 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:30.394+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:35.396 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:35.394+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:35.501 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~124s 2024-09-22T03:54:40.396 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:40.394+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:40.904 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-09-22T03:54:45.396 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:45.396+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:46.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-09-22T03:54:50.397 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:50.396+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:51.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~141s 2024-09-22T03:54:55.397 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:54:55.396+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:54:57.112 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-09-22T03:55:00.397 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:00.395+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:02.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-09-22T03:55:05.398 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:05.396+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:07.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~157s 2024-09-22T03:55:10.398 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:10.396+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:13.322 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~162s 2024-09-22T03:55:15.398 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:15.397+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:18.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~168s 2024-09-22T03:55:20.399 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:20.397+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:24.127 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~173s 2024-09-22T03:55:25.399 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:25.397+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:29.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~178s 2024-09-22T03:55:30.400 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:30.399+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:34.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~184s 2024-09-22T03:55:35.400 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:35.399+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:40.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~189s 2024-09-22T03:55:40.400 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:40.399+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:45.401 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:45.399+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:45.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-09-22T03:55:50.401 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:50.399+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:51.143 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-09-22T03:55:55.401 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:55:55.400+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:55:56.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-09-22T03:56:00.402 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:00.400+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:01.949 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~211s 2024-09-22T03:56:05.402 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:05.401+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:07.352 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~216s 2024-09-22T03:56:10.403 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:10.402+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:12.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~222s 2024-09-22T03:56:15.403 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:15.402+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:18.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~227s 2024-09-22T03:56:20.404 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:20.403+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:23.561 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~232s 2024-09-22T03:56:25.404 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:25.402+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:28.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~238s 2024-09-22T03:56:30.405 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:30.403+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:34.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~243s 2024-09-22T03:56:35.405 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:35.403+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:39.769 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2024-09-22T03:56:40.405 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:40.404+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:45.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~254s 2024-09-22T03:56:45.406 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:45.404+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:50.406 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:50.404+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:50.575 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2024-09-22T03:56:55.407 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:56:55.406+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:56:55.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-09-22T03:57:00.409 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:00.406+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:01.381 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-09-22T03:57:05.409 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:05.407+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:06.783 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-09-22T03:57:10.410 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:10.408+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:12.186 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-09-22T03:57:15.410 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:15.408+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:17.589 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-09-22T03:57:20.410 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:20.409+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:22.993 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2024-09-22T03:57:25.411 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:25.409+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:28.395 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2024-09-22T03:57:30.411 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:30.409+0000 7fdc6cdf9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.83:6803/4026345099 for 29sec e26 v26) 2024-09-22T03:57:33.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2024-09-22T03:57:33.798 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-09-22T03:57:34.201 INFO:tasks.ceph.osd.0:Sent signal 15 2024-09-22T03:57:34.201 INFO:tasks.ceph.osd.2:Sent signal 15 2024-09-22T03:57:34.201 INFO:tasks.ceph.mon.a:Sent signal 15 2024-09-22T03:57:34.202 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-09-22T03:57:34.202 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:34.201+0000 7fdc6fdff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 28677) UID: 0 2024-09-22T03:57:34.202 INFO:tasks.ceph.mon.a.smithi083.stderr:2024-09-22T03:57:34.202+0000 7fdc6fdff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-22T03:57:34.203 INFO:tasks.ceph.mgr.x.smithi083.stderr:2024-09-22T03:57:34.202+0000 7f5c877f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 28680) UID: 0 2024-09-22T11:39:00.480 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-22T11:39:00.533 DEBUG:teuthology.task.console_log:Killing console logger for smithi083 2024-09-22T11:39:00.534 DEBUG:teuthology.exit:Finished running handlers