2024-09-22T03:31:18.202 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:31:18.222 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:31:18.304 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914365 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_fio} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914365' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ90l0Km5eIPvnLbI3DjNxDsdOXrpDEa8h1PGIv3rgvP3xGXbjeU6S5YgSgJ/S8FhGIXpwV1HTFATn2l/oqKu9s= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randwrite norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:31:18.304 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-22T03:31:18.305 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-22T03:31:18.305 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:31:18.306 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:31:18.327 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-22T03:31:18.327 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:31:18.327 INFO:teuthology.packaging:ref: None 2024-09-22T03:31:18.327 INFO:teuthology.packaging:tag: None 2024-09-22T03:31:18.327 INFO:teuthology.packaging:branch: main 2024-09-22T03:31:18.327 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:31:18.327 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-22T03:31:18.545 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-22T03:31:18.547 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:31:18.553 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:31:18.553 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:31:18.621 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:31:18.639 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:31:18.653 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:31:18.677 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914365', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:26:19.450537', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ90l0Km5eIPvnLbI3DjNxDsdOXrpDEa8h1PGIv3rgvP3xGXbjeU6S5YgSgJ/S8FhGIXpwV1HTFATn2l/oqKu9s='} 2024-09-22T03:31:18.677 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:31:18.727 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:31:18.727 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:31:18.801 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6cfa7f4160>, signals=[15]) 2024-09-22T03:31:18.801 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:31:18.829 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:31:18.829 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-09-22T03:31:18.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:31:18.923 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:31:18.929 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-09-22T03:31:19.010 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-09-22T03:31:19.011 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:31:19.066 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:31:19.067 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:31:19.067 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-09-22T03:31:19.179 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:31:19.186 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:31:19.283 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:31:19.284 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:31:19.301 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:31:19.356 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:31:19.357 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:31:19.447 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:31:19.494 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:31:19.507 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:31:19.507 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:31:19.508 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:31:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:31:19.526 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-22T03:31:19.583 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:31:19.584 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-503.el9.x86_64 2024-09-22T03:31:19.584 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-09-22T03:31:21.727 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-22T03:31:24.578 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 12 MB/s | 20 MB 00:01 2024-09-22T03:31:28.837 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-22T03:31:30.738 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 19 kB 00:00 2024-09-22T03:31:32.550 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-22T03:31:38.022 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-22T03:31:39.954 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:31:39.954 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:31:40.051 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:31:40.060 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:31:40.061 INFO:teuthology.orchestra.run.smithi174.stdout:Install 4 Packages 2024-09-22T03:31:40.061 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:31:40.061 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 79 M 2024-09-22T03:31:40.061 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 126 M 2024-09-22T03:31:40.061 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:31:40.282 INFO:teuthology.orchestra.run.smithi174.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 431 kB/s | 36 kB 00:00 2024-09-22T03:31:40.657 INFO:teuthology.orchestra.run.smithi174.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2024-09-22T03:31:41.015 INFO:teuthology.orchestra.run.smithi174.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 39 MB/s | 29 MB 00:00 2024-09-22T03:31:41.383 INFO:teuthology.orchestra.run.smithi174.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-22T03:31:41.383 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:31:41.383 INFO:teuthology.orchestra.run.smithi174.stdout:Total 60 MB/s | 79 MB 00:01 2024-09-22T03:31:42.065 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:31:42.421 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:31:42.422 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:31:43.412 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:31:43.412 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:31:45.839 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:31:46.526 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:31:46.609 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:31:49.358 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:31:49.476 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:31:54.314 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:31:54.435 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:31:59.300 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:32:37.455 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:32:37.501 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:32:37.850 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:32:37.850 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:32:37.850 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:32:37.850 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:32:38.694 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:32:38.694 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:32:38.694 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:32:38.694 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:32:38.695 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:32:38.695 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:32:38.695 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:32:38.695 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:32:38.695 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:32:39.022 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:32:39.142 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:32:39.142 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:32:39.142 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:32:39.142 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:32:39.142 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:32:39.143 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:32:39.143 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:32:39.143 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:32:39.202 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:32:39.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:32:39.202 DEBUG:teuthology.orchestra.run.smithi174:> sudo rpm -qi grub2-tools 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Name : grub2-tools 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Epoch : 1 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Version : 2.06 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Release : 82.el9 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: x86_64 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Group : Unspecified 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Size : 8274670 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:License : GPLv3+ 2024-09-22T03:32:39.239 INFO:teuthology.orchestra.run.smithi174.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Packager : builder@centos.org 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Vendor : CentOS 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Summary : Support tools for GRUB. 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:Description : 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:hardware devices. 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:32:39.240 INFO:teuthology.orchestra.run.smithi174.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:32:39.241 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:32:39.241 INFO:teuthology.task.kernel:Updating grub on smithi174 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:32:39.241 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:32:39.887 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-09-22T03:32:41.115 INFO:teuthology.orchestra.run.smithi174.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:32:41.142 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-09-22T03:32:41.144 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-09-22T03:32:41.160 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.MzC9Gs5IbZ 2024-09-22T03:32:41.160 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MzC9Gs5IbZ 2024-09-22T03:32:41.227 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.MzC9Gs5IbZ 2024-09-22T03:32:41.367 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.MzC9Gs5IbZ is 6KB 2024-09-22T03:32:41.415 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.MzC9Gs5IbZ 2024-09-22T03:32:41.429 DEBUG:teuthology.orchestra.run.smithi174:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:32:41.495 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:32:41.495 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:32:41.495 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:32:41.495 INFO:teuthology.orchestra.run.smithi174.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:32:41.496 DEBUG:teuthology.orchestra.run.smithi174:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:32:41.657 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-09-22T03:33:11.688 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:33:11.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:33:11.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:33:30.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-22T03:33:39.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:33:39.232 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:33:42.287 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-22T03:33:54.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:33:54.297 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:34:12.719 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-09-22T03:34:27.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:34:27.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:34:28.092 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-22T03:34:28.633 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:34:28.646 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:34:28.647 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-09-22T03:34:28.683 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:34:28.683 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:34:28.683 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:34:28.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:34:29.684 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:34:29.684 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:34:29.737 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-09-22T03:34:29.913 DEBUG:teuthology.parallel:result is None 2024-09-22T03:34:29.913 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:34:29.922 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:34:29.922 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:34:29.937 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:34:29.990 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:34:30.040 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:34:30.041 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:34:30.087 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:34:30.094 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:34:30.094 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:34:30.172 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:34:30.182 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:34:30.184 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:34:30.190 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:34:30.191 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:34:30.215 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:34:30.225 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:34:30.225 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:34:30.270 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:34:30.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:34:30.397 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:34:30.476 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:34:30.519 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:34:30.519 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:34:30.583 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-09-22T03:34:30.672 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:34:31.027 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:34:31.034 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:34:31.034 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:34:31.082 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:34:31.112 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-09-22T03:34:31.160 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-09-22T03:34:31.162 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:34:31.162 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-09-22T03:34:31.187 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-09-22T03:34:31.188 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:34:31.188 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T03:34:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:34:31.253 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:34:31.253 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:34:31.329 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:34:31.338 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-22T03:34:31.340 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-22T03:34:31.443 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:34:31.452 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:34:31.453 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:38:10.024 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-09-22T03:38:10.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:38:10.025 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:38:10.100 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-09-22T03:38:10.178 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-09-22T03:38:10.178 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:38:10.189 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:38:10.190 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:38:10.190 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:38:10.246 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:38:10.260 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:38:10.284 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-09-22T03:38:10.295 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-09-22T03:38:10.309 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:38:10.323 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:38:10.380 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:38:10.384 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:38:10.385 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:38:10.393 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:38:10.393 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-22T03:38:10.393 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-22T03:38:10.393 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:38:10.398 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:38:10.398 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:38:10.398 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-22T03:38:10.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:38:10.546 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-22T03:38:10.546 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-22T03:38:10.752 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:38:10.752 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:38:10.752 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:38:10.778 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:38:10.778 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:38:10.844 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:38:10.919 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:38:10.983 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-09-22T03:38:10.984 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-09-22T03:38:11.209 INFO:teuthology.orchestra.run.smithi174.stdout:45 files removed 2024-09-22T03:38:11.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw 2024-09-22T03:38:11.805 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 307 kB/s | 88 kB 00:00 2024-09-22T03:38:12.098 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-22T03:38:12.373 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 8.7 kB/s | 2.1 kB 00:00 2024-09-22T03:38:15.665 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.3 MB 00:03 2024-09-22T03:38:24.548 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-22T03:38:29.616 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.5 MB 00:01 2024-09-22T03:38:31.241 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 19 kB 00:00 2024-09-22T03:38:33.045 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 13 MB/s | 23 MB 00:01 2024-09-22T03:38:38.418 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-22T03:38:40.628 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:38:40.645 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:38:40.646 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:38:40.647 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Install 31 Packages 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 2 Packages 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 60 M 2024-09-22T03:38:40.648 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:38:41.577 INFO:teuthology.orchestra.run.smithi174.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 8.1 MB/s | 6.1 MB 00:00 2024-09-22T03:38:41.644 INFO:teuthology.orchestra.run.smithi174.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 378 kB/s | 25 kB 00:00 2024-09-22T03:38:41.778 INFO:teuthology.orchestra.run.smithi174.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 5.9 MB/s | 808 kB 00:00 2024-09-22T03:38:41.911 INFO:teuthology.orchestra.run.smithi174.stdout:(4/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 9.5 MB/s | 10 MB 00:01 2024-09-22T03:38:41.953 INFO:teuthology.orchestra.run.smithi174.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 3.1 MB/s | 558 kB 00:00 2024-09-22T03:38:42.020 INFO:teuthology.orchestra.run.smithi174.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 678 kB/s | 45 kB 00:00 2024-09-22T03:38:42.087 INFO:teuthology.orchestra.run.smithi174.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 2.2 MB/s | 148 kB 00:00 2024-09-22T03:38:42.154 INFO:teuthology.orchestra.run.smithi174.stdout:(8/33): python3-cephfs-19.3.0-5092.g62da54b4.el 2.3 MB/s | 159 kB 00:00 2024-09-22T03:38:42.229 INFO:teuthology.orchestra.run.smithi174.stdout:(9/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_6 17 MB/s | 5.4 MB 00:00 2024-09-22T03:38:42.254 INFO:teuthology.orchestra.run.smithi174.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 3.1 MB/s | 314 kB 00:00 2024-09-22T03:38:42.288 INFO:teuthology.orchestra.run.smithi174.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-22T03:38:42.305 INFO:teuthology.orchestra.run.smithi174.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.9 MB/s | 99 kB 00:00 2024-09-22T03:38:42.497 INFO:teuthology.orchestra.run.smithi174.stdout:(13/33): ceph-common-19.3.0-5092.g62da54b4.el9. 12 MB/s | 20 MB 00:01 2024-09-22T03:38:42.630 INFO:teuthology.orchestra.run.smithi174.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 312 kB/s | 107 kB 00:00 2024-09-22T03:38:42.655 INFO:teuthology.orchestra.run.smithi174.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 487 kB/s | 171 kB 00:00 2024-09-22T03:38:42.681 INFO:teuthology.orchestra.run.smithi174.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 873 kB/s | 160 kB 00:00 2024-09-22T03:38:42.706 INFO:teuthology.orchestra.run.smithi174.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 601 kB/s | 45 kB 00:00 2024-09-22T03:38:42.756 INFO:teuthology.orchestra.run.smithi174.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 850 kB/s | 42 kB 00:00 2024-09-22T03:38:42.782 INFO:teuthology.orchestra.run.smithi174.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-22T03:38:42.815 INFO:teuthology.orchestra.run.smithi174.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-22T03:38:42.841 INFO:teuthology.orchestra.run.smithi174.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 614 kB/s | 36 kB 00:00 2024-09-22T03:38:42.874 INFO:teuthology.orchestra.run.smithi174.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-22T03:38:42.899 INFO:teuthology.orchestra.run.smithi174.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-09-22T03:38:42.950 INFO:teuthology.orchestra.run.smithi174.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-22T03:38:42.975 INFO:teuthology.orchestra.run.smithi174.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 383 kB/s | 26 kB 00:00 2024-09-22T03:38:42.991 INFO:teuthology.orchestra.run.smithi174.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-09-22T03:38:43.017 INFO:teuthology.orchestra.run.smithi174.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-22T03:38:43.067 INFO:teuthology.orchestra.run.smithi174.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-22T03:38:43.142 INFO:teuthology.orchestra.run.smithi174.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 839 kB 00:00 2024-09-22T03:38:43.309 INFO:teuthology.orchestra.run.smithi174.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-09-22T03:38:43.384 INFO:teuthology.orchestra.run.smithi174.stdout:(31/33): librados2-19.3.0-5092.g62da54b4.el9.x8 14 MB/s | 3.5 MB 00:00 2024-09-22T03:38:43.476 INFO:teuthology.orchestra.run.smithi174.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-09-22T03:38:43.585 INFO:teuthology.orchestra.run.smithi174.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-22T03:38:43.590 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:38:43.590 INFO:teuthology.orchestra.run.smithi174.stdout:Total 20 MB/s | 60 MB 00:02 2024-09-22T03:38:43.704 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:38:43.740 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:38:43.740 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:38:44.404 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:38:44.405 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:38:45.898 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:38:46.423 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-22T03:38:46.470 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:38:46.562 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:38:46.652 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:38:46.680 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:38:46.743 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:38:46.789 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-22T03:38:46.818 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-22T03:38:46.843 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-22T03:38:46.894 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:38:46.915 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T03:38:47.127 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:38:47.157 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T03:38:47.192 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-22T03:38:47.227 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-22T03:38:47.256 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:38:47.336 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-22T03:38:47.361 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:38:47.442 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T03:38:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-22T03:38:47.593 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-22T03:38:47.635 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-22T03:38:47.677 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-22T03:38:47.709 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-22T03:38:47.738 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-22T03:38:47.778 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-22T03:38:48.143 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-22T03:38:48.210 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:38:48.301 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T03:38:48.331 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-22T03:38:48.367 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-22T03:38:48.955 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-22T03:38:49.079 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:38:49.150 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T03:38:49.232 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-22T03:38:49.264 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-22T03:38:51.748 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:38:51.785 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:38:52.488 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T03:38:52.530 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:38:52.604 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T03:38:52.604 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:38:52.604 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:38:52.604 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:38:52.648 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-09-22T03:39:02.944 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:04.053 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:39:04.079 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T03:39:04.079 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:39:04.079 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:39:04.079 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:39:04.079 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:39:04.080 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:04.081 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:39:04.164 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T03:39:04.164 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:39:05.445 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:39:05.445 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-22T03:39:05.446 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-22T03:39:05.447 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-22T03:39:05.448 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-22T03:39:05.448 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-22T03:39:05.448 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T03:39:05.448 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-22T03:39:05.448 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-22T03:39:06.471 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T03:39:06.471 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:06.471 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-09-22T03:39:06.471 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:39:06.472 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:39:06.473 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:39:06.474 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:39:06.474 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:39:06.474 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:06.474 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:39:06.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-test 2024-09-22T03:39:07.403 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:29 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:39:07.534 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:39:07.535 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout:Install 9 Packages 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:07.536 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 83 M 2024-09-22T03:39:07.537 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 775 M 2024-09-22T03:39:07.539 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:39:07.726 INFO:teuthology.orchestra.run.smithi174.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 833 kB/s | 94 kB 00:00 2024-09-22T03:39:07.859 INFO:teuthology.orchestra.run.smithi174.stdout:(2/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 709 kB/s | 175 kB 00:00 2024-09-22T03:39:07.951 INFO:teuthology.orchestra.run.smithi174.stdout:(3/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 2.2 MB/s | 513 kB 00:00 2024-09-22T03:39:08.060 INFO:teuthology.orchestra.run.smithi174.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 912 kB/s | 183 kB 00:00 2024-09-22T03:39:08.202 INFO:teuthology.orchestra.run.smithi174.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-22T03:39:08.252 INFO:teuthology.orchestra.run.smithi174.stdout:(6/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-22T03:39:08.285 INFO:teuthology.orchestra.run.smithi174.stdout:(7/9): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-09-22T03:39:08.352 INFO:teuthology.orchestra.run.smithi174.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.2 MB/s | 122 kB 00:00 2024-09-22T03:39:10.128 INFO:teuthology.orchestra.run.smithi174.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 32 MB/s | 81 MB 00:02 2024-09-22T03:39:10.131 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:39:10.131 INFO:teuthology.orchestra.run.smithi174.stdout:Total 32 MB/s | 83 MB 00:02 2024-09-22T03:39:10.217 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:39:10.239 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:39:10.239 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:39:10.832 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:39:10.832 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:39:11.722 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:39:11.785 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-22T03:39:11.845 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-22T03:39:11.898 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-22T03:39:11.930 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-22T03:39:11.965 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:39:11.989 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:39:12.059 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T03:39:12.106 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-22T03:39:12.133 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:39:22.930 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T03:39:23.005 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:39:28.737 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-22T03:39:28.738 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-22T03:39:29.171 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-22T03:39:29.171 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:29.171 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:39:29.171 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:29.171 INFO:teuthology.orchestra.run.smithi174.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:29.172 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:39:29.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph 2024-09-22T03:39:30.059 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:52 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:39:30.194 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:39:30.196 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:39:30.197 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:39:30.198 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:39:30.199 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:Install 37 Packages 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 84 M 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 896 M 2024-09-22T03:39:30.200 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:39:30.559 INFO:teuthology.orchestra.run.smithi174.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 59 kB/s | 6.5 kB 00:00 2024-09-22T03:39:30.777 INFO:teuthology.orchestra.run.smithi174.stdout:(2/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 6.7 MB/s | 2.2 MB 00:00 2024-09-22T03:39:30.810 INFO:teuthology.orchestra.run.smithi174.stdout:(3/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 4.4 MB/s | 1.6 MB 00:00 2024-09-22T03:39:30.885 INFO:teuthology.orchestra.run.smithi174.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 3.8 MB/s | 292 kB 00:00 2024-09-22T03:39:31.011 INFO:teuthology.orchestra.run.smithi174.stdout:(5/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 11 MB/s | 5.1 MB 00:00 2024-09-22T03:39:31.070 INFO:teuthology.orchestra.run.smithi174.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-22T03:39:31.161 INFO:teuthology.orchestra.run.smithi174.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 8.2 MB/s | 1.2 MB 00:00 2024-09-22T03:39:31.186 INFO:teuthology.orchestra.run.smithi174.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 911 kB/s | 106 kB 00:00 2024-09-22T03:39:31.228 INFO:teuthology.orchestra.run.smithi174.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-22T03:39:31.262 INFO:teuthology.orchestra.run.smithi174.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-09-22T03:39:31.312 INFO:teuthology.orchestra.run.smithi174.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-22T03:39:31.479 INFO:teuthology.orchestra.run.smithi174.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 869 kB/s | 188 kB 00:00 2024-09-22T03:39:31.504 INFO:teuthology.orchestra.run.smithi174.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 898 kB/s | 172 kB 00:00 2024-09-22T03:39:31.529 INFO:teuthology.orchestra.run.smithi174.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 694 kB/s | 35 kB 00:00 2024-09-22T03:39:31.546 INFO:teuthology.orchestra.run.smithi174.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-22T03:39:31.580 INFO:teuthology.orchestra.run.smithi174.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-22T03:39:31.605 INFO:teuthology.orchestra.run.smithi174.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 712 kB/s | 42 kB 00:00 2024-09-22T03:39:31.647 INFO:teuthology.orchestra.run.smithi174.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.5 MB/s | 151 kB 00:00 2024-09-22T03:39:31.672 INFO:teuthology.orchestra.run.smithi174.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-22T03:39:31.698 INFO:teuthology.orchestra.run.smithi174.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.7 MB/s | 172 kB 00:00 2024-09-22T03:39:31.739 INFO:teuthology.orchestra.run.smithi174.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 8.4 MB/s | 358 kB 00:00 2024-09-22T03:39:31.898 INFO:teuthology.orchestra.run.smithi174.stdout:(22/37): lua-devel-5.4.4-4.el9.x86_64.rpm 70 kB/s | 22 kB 00:00 2024-09-22T03:39:31.949 INFO:teuthology.orchestra.run.smithi174.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 355 kB/s | 18 kB 00:00 2024-09-22T03:39:31.974 INFO:teuthology.orchestra.run.smithi174.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 936 kB/s | 23 kB 00:00 2024-09-22T03:39:31.999 INFO:teuthology.orchestra.run.smithi174.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 41 kB/s | 11 kB 00:00 2024-09-22T03:39:32.032 INFO:teuthology.orchestra.run.smithi174.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 736 kB/s | 20 kB 00:00 2024-09-22T03:39:32.058 INFO:teuthology.orchestra.run.smithi174.stdout:(27/37): python3-logutils-0.3.5-21.el9.noarch.r 1.9 MB/s | 46 kB 00:00 2024-09-22T03:39:32.083 INFO:teuthology.orchestra.run.smithi174.stdout:(28/37): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-09-22T03:39:32.108 INFO:teuthology.orchestra.run.smithi174.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-22T03:39:32.134 INFO:teuthology.orchestra.run.smithi174.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-22T03:39:32.167 INFO:teuthology.orchestra.run.smithi174.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-09-22T03:39:32.192 INFO:teuthology.orchestra.run.smithi174.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-22T03:39:32.217 INFO:teuthology.orchestra.run.smithi174.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-22T03:39:32.243 INFO:teuthology.orchestra.run.smithi174.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 4.4 MB/s | 230 kB 00:00 2024-09-22T03:39:32.269 INFO:teuthology.orchestra.run.smithi174.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-22T03:39:32.302 INFO:teuthology.orchestra.run.smithi174.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-22T03:39:32.886 INFO:teuthology.orchestra.run.smithi174.stdout:(37/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 33 MB/s | 70 MB 00:02 2024-09-22T03:39:32.890 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:39:32.890 INFO:teuthology.orchestra.run.smithi174.stdout:Total 31 MB/s | 84 MB 00:02 2024-09-22T03:39:33.045 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:39:33.087 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:39:33.087 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:39:33.875 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:39:33.876 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:39:35.629 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:39:35.696 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-22T03:39:35.729 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-22T03:39:35.757 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-22T03:39:35.828 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-22T03:39:35.881 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-22T03:39:35.913 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-22T03:39:36.005 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-22T03:39:36.084 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-22T03:39:36.124 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-22T03:39:36.159 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-22T03:39:36.189 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-22T03:39:36.230 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-22T03:39:36.264 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-22T03:39:36.304 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-22T03:39:36.339 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-22T03:39:36.373 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-22T03:39:36.424 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-22T03:39:36.456 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-22T03:39:36.529 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-22T03:39:36.676 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-22T03:39:36.772 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-22T03:39:36.825 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-22T03:39:36.864 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-22T03:39:36.921 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-22T03:39:36.975 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-22T03:39:37.161 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-22T03:39:37.217 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-22T03:39:37.295 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-22T03:39:37.383 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-22T03:39:37.429 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-22T03:39:37.559 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-22T03:39:37.753 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-22T03:39:37.785 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:39:37.811 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T03:39:37.812 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:39:37.812 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:39:37.812 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:39:37.812 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:39:37.812 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:48.302 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:39:48.329 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:48.877 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:39:48.901 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T03:39:48.901 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:39:48.901 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:39:48.901 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:39:48.901 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:39:48.902 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:49.136 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:39:49.160 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:49.233 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-22T03:39:55.628 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-22T03:39:55.629 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-22T03:39:55.630 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-22T03:39:55.631 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-22T03:39:55.631 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-22T03:39:55.631 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-22T03:39:55.631 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-22T03:39:55.632 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-22T03:39:56.756 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:39:56.757 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:39:56.758 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:39:56.759 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:56.760 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:39:57.053 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-base 2024-09-22T03:39:57.656 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:19 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:39:57.752 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:39:57.797 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:39:57.798 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:39:57.798 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:39:57.882 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install cephadm 2024-09-22T03:39:58.469 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:20 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:39:58.600 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-22T03:39:58.601 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 784 k 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 791 k 2024-09-22T03:39:58.602 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:39:58.869 INFO:teuthology.orchestra.run.smithi174.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 2.9 MB/s | 784 kB 00:00 2024-09-22T03:39:58.869 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:39:58.870 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-22T03:39:58.870 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:39:58.875 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:39:58.875 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:39:58.881 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:39:58.881 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:39:58.950 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:39:59.331 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:39:59.430 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:39:59.803 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:00.054 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:00.280 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:40:00.868 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:22 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:01.002 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 158 k 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 644 k 2024-09-22T03:40:01.003 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:01.215 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 748 kB/s | 158 kB 00:00 2024-09-22T03:40:01.216 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:01.216 INFO:teuthology.orchestra.run.smithi174.stdout:Total 741 kB/s | 158 kB 00:00 2024-09-22T03:40:01.216 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:01.226 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:01.226 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:01.284 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:01.285 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:01.437 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:01.533 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:40:01.557 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:40:01.557 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:40:01.558 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:40:01.558 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:02.474 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:02.720 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr 2024-09-22T03:40:03.309 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:25 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:03.400 INFO:teuthology.orchestra.run.smithi174.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:03.443 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:03.443 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:03.444 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:03.522 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:40:04.104 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:26 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:04.237 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-22T03:40:04.238 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout:Install 11 Packages 2024-09-22T03:40:04.239 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:04.240 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 18 M 2024-09-22T03:40:04.240 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 156 M 2024-09-22T03:40:04.240 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:04.918 INFO:teuthology.orchestra.run.smithi174.stdout:(1/11): ceph-prometheus-alerts-19.3.0-5092.g62d 106 kB/s | 16 kB 00:00 2024-09-22T03:40:04.943 INFO:teuthology.orchestra.run.smithi174.stdout:(2/11): ceph-grafana-dashboards-19.3.0-5092.g62 190 kB/s | 34 kB 00:00 2024-09-22T03:40:05.068 INFO:teuthology.orchestra.run.smithi174.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-22T03:40:05.136 INFO:teuthology.orchestra.run.smithi174.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.1 MB/s | 548 kB 00:00 2024-09-22T03:40:05.161 INFO:teuthology.orchestra.run.smithi174.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 786 kB/s | 19 kB 00:00 2024-09-22T03:40:05.278 INFO:teuthology.orchestra.run.smithi174.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-09-22T03:40:05.303 INFO:teuthology.orchestra.run.smithi174.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-22T03:40:05.328 INFO:teuthology.orchestra.run.smithi174.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-22T03:40:05.353 INFO:teuthology.orchestra.run.smithi174.stdout:(9/11): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-22T03:40:05.495 INFO:teuthology.orchestra.run.smithi174.stdout:(10/11): ceph-mgr-dashboard-19.3.0-5092.g62da54 19 MB/s | 14 MB 00:00 2024-09-22T03:40:05.687 INFO:teuthology.orchestra.run.smithi174.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-09-22T03:40:05.691 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:05.691 INFO:teuthology.orchestra.run.smithi174.stdout:Total 13 MB/s | 18 MB 00:01 2024-09-22T03:40:05.725 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:05.745 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:05.745 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:06.011 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:06.011 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:06.562 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:06.652 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:40:06.690 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:40:06.789 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:40:06.916 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:40:06.968 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:40:07.060 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:40:07.111 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:40:07.140 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:40:07.214 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-22T03:40:10.517 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-22T03:40:10.584 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:40:12.284 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T03:40:12.284 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-22T03:40:12.284 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:40:12.285 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:40:12.596 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:12.597 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:12.862 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:40:13.433 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:35 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:13.567 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:13.568 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-09-22T03:40:13.568 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:13.568 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:40:13.569 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout:Install 12 Packages 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:13.570 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 38 M 2024-09-22T03:40:13.571 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 212 M 2024-09-22T03:40:13.571 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:14.408 INFO:teuthology.orchestra.run.smithi174.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 257 kB/s | 191 kB 00:00 2024-09-22T03:40:14.533 INFO:teuthology.orchestra.run.smithi174.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 263 kB/s | 33 kB 00:00 2024-09-22T03:40:14.558 INFO:teuthology.orchestra.run.smithi174.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 896 kB/s | 801 kB 00:00 2024-09-22T03:40:14.667 INFO:teuthology.orchestra.run.smithi174.stdout:(4/12): ceph-mgr-diskprediction-local-19.3.0-50 7.4 MB/s | 7.4 MB 00:01 2024-09-22T03:40:14.792 INFO:teuthology.orchestra.run.smithi174.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 78 kB/s | 18 kB 00:00 2024-09-22T03:40:14.817 INFO:teuthology.orchestra.run.smithi174.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 256 kB/s | 38 kB 00:00 2024-09-22T03:40:14.968 INFO:teuthology.orchestra.run.smithi174.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-09-22T03:40:15.251 INFO:teuthology.orchestra.run.smithi174.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.2 MB/s | 3.0 MB 00:00 2024-09-22T03:40:15.410 INFO:teuthology.orchestra.run.smithi174.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.5 MB/s | 250 kB 00:00 2024-09-22T03:40:16.627 INFO:teuthology.orchestra.run.smithi174.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.7 MB/s | 4.9 MB 00:01 2024-09-22T03:40:16.960 INFO:teuthology.orchestra.run.smithi174.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.6 MB/s | 5.1 MB 00:01 2024-09-22T03:40:18.452 INFO:teuthology.orchestra.run.smithi174.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 5.3 MB/s | 16 MB 00:03 2024-09-22T03:40:18.453 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:18.453 INFO:teuthology.orchestra.run.smithi174.stdout:Total 7.8 MB/s | 38 MB 00:04 2024-09-22T03:40:18.807 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:18.834 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:18.834 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:19.258 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:19.259 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:20.017 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:20.092 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:40:20.133 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:40:20.231 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:40:20.268 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:40:20.443 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:40:20.483 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:40:20.559 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:40:21.209 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:40:21.329 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:40:22.580 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:40:23.620 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:40:23.656 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:40:26.213 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T03:40:26.213 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-22T03:40:26.213 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:40:26.213 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:40:26.213 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:40:26.214 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:40:26.618 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:26.619 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:27.009 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:40:27.637 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:49 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:27.775 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:40:27.776 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout:Install 13 Packages 2024-09-22T03:40:27.777 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:27.778 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.2 M 2024-09-22T03:40:27.778 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 26 M 2024-09-22T03:40:27.778 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:28.280 INFO:teuthology.orchestra.run.smithi174.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 224 kB/s | 26 kB 00:00 2024-09-22T03:40:28.305 INFO:teuthology.orchestra.run.smithi174.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 137 kB/s | 19 kB 00:00 2024-09-22T03:40:28.330 INFO:teuthology.orchestra.run.smithi174.stdout:(3/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 298 kB/s | 49 kB 00:00 2024-09-22T03:40:28.414 INFO:teuthology.orchestra.run.smithi174.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-22T03:40:28.439 INFO:teuthology.orchestra.run.smithi174.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-22T03:40:28.473 INFO:teuthology.orchestra.run.smithi174.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 916 kB/s | 54 kB 00:00 2024-09-22T03:40:28.506 INFO:teuthology.orchestra.run.smithi174.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-22T03:40:28.532 INFO:teuthology.orchestra.run.smithi174.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-22T03:40:28.557 INFO:teuthology.orchestra.run.smithi174.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-22T03:40:28.591 INFO:teuthology.orchestra.run.smithi174.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-22T03:40:28.616 INFO:teuthology.orchestra.run.smithi174.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-22T03:40:28.641 INFO:teuthology.orchestra.run.smithi174.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-22T03:40:28.708 INFO:teuthology.orchestra.run.smithi174.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-09-22T03:40:28.711 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:28.711 INFO:teuthology.orchestra.run.smithi174.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-22T03:40:28.771 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:28.786 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:28.786 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:28.875 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:28.875 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:29.381 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:29.546 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:40:29.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:40:29.656 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:40:29.696 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:40:29.734 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:40:29.836 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:40:29.938 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:40:29.994 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:40:30.713 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:40:30.766 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:40:30.800 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:40:30.848 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:40:30.870 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:40:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T03:40:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-22T03:40:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:40:31.625 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:40:31.626 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:40:31.970 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:40:31.970 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:31.970 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:31.970 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:40:31.971 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:31.972 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:32.240 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:40:32.821 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:54 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:32.954 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:32.955 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:32.955 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:32.955 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout:Install 6 Packages 2024-09-22T03:40:32.956 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:32.957 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 7.0 M 2024-09-22T03:40:32.957 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 33 M 2024-09-22T03:40:32.957 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:33.384 INFO:teuthology.orchestra.run.smithi174.stdout:(1/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 709 kB/s | 154 kB 00:00 2024-09-22T03:40:33.409 INFO:teuthology.orchestra.run.smithi174.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.0 MB/s | 248 kB 00:00 2024-09-22T03:40:33.467 INFO:teuthology.orchestra.run.smithi174.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-22T03:40:33.493 INFO:teuthology.orchestra.run.smithi174.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 690 kB/s | 58 kB 00:00 2024-09-22T03:40:33.518 INFO:teuthology.orchestra.run.smithi174.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-22T03:40:33.660 INFO:teuthology.orchestra.run.smithi174.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-22T03:40:33.662 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:33.662 INFO:teuthology.orchestra.run.smithi174.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-22T03:40:33.732 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:33.743 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:33.744 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:33.844 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:33.845 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:34.147 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:34.318 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:40:34.378 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:40:34.888 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:40:34.972 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:40:35.040 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:40:35.065 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:40:35.826 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:40:36.083 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:40:36.084 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:40:36.084 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:40:36.084 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:36.084 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:36.349 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-fuse 2024-09-22T03:40:36.930 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:01:58 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:37.062 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-22T03:40:37.063 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout:Install 2 Packages 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 963 k 2024-09-22T03:40:37.064 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 4.1 M 2024-09-22T03:40:37.065 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:37.378 INFO:teuthology.orchestra.run.smithi174.stdout:(1/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.0 MB/s | 883 kB 00:00 2024-09-22T03:40:37.728 INFO:teuthology.orchestra.run.smithi174.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 124 kB/s | 80 kB 00:00 2024-09-22T03:40:37.728 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:37.729 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.4 MB/s | 963 kB 00:00 2024-09-22T03:40:37.740 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:37.765 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:37.766 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:37.845 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:37.846 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:38.032 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:38.182 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:40:38.243 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:40:38.999 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T03:40:38.999 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-22T03:40:39.264 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:40:39.264 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:39.264 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:39.265 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:40:39.265 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:39.265 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:39.558 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-volume 2024-09-22T03:40:40.152 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:02 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:40.287 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 285 k 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 1.3 M 2024-09-22T03:40:40.288 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:40.658 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 775 kB/s | 285 kB 00:00 2024-09-22T03:40:40.659 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:40.659 INFO:teuthology.orchestra.run.smithi174.stdout:Total 771 kB/s | 285 kB 00:00 2024-09-22T03:40:40.659 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:40.666 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:40.666 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:40.696 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:40.696 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:40.915 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:40.975 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:40:40.991 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:40:40.991 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:40:40.991 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:40:40.992 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:41.974 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:42.276 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados-devel 2024-09-22T03:40:42.878 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:04 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:43.012 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:43.013 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:43.013 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-22T03:40:43.013 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:43.013 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:43.013 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 125 k 2024-09-22T03:40:43.014 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 445 k 2024-09-22T03:40:43.015 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:43.286 INFO:teuthology.orchestra.run.smithi174.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 458 kB/s | 125 kB 00:00 2024-09-22T03:40:43.287 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:43.287 INFO:teuthology.orchestra.run.smithi174.stdout:Total 455 kB/s | 125 kB 00:00 2024-09-22T03:40:43.287 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:43.292 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:43.292 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:43.357 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:43.357 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:43.507 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:43.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:44.044 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:44.309 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:44.309 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:44.309 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:44.309 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:44.310 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:44.310 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:44.570 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs2 2024-09-22T03:40:45.161 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:07 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:45.251 INFO:teuthology.orchestra.run.smithi174.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:45.294 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:45.295 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:45.296 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:45.372 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install libcephfs-devel 2024-09-22T03:40:45.951 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:07 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:46.084 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:46.084 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:46.085 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:46.086 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 32 k 2024-09-22T03:40:46.086 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 142 k 2024-09-22T03:40:46.086 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:46.230 INFO:teuthology.orchestra.run.smithi174.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 221 kB/s | 32 kB 00:00 2024-09-22T03:40:46.230 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:46.231 INFO:teuthology.orchestra.run.smithi174.stdout:Total 219 kB/s | 32 kB 00:00 2024-09-22T03:40:46.231 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:46.234 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:46.234 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:46.257 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:46.257 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:46.344 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:46.452 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:46.860 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:47.089 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:47.089 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:47.089 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:47.089 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:47.090 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:47.090 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:47.292 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librados2 2024-09-22T03:40:47.868 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:47.957 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:48.000 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:48.001 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:48.002 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:48.078 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install librbd1 2024-09-22T03:40:48.653 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:10 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:48.742 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:48.786 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:48.787 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:48.787 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:48.870 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rados 2024-09-22T03:40:49.447 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:11 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:49.536 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:49.580 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:49.581 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:49.581 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:49.665 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rgw 2024-09-22T03:40:50.241 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:12 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:50.330 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:50.373 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:50.375 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:50.375 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:50.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-cephfs 2024-09-22T03:40:51.035 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:13 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:51.127 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:51.170 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:51.171 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:51.171 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:51.253 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-rbd 2024-09-22T03:40:51.829 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:13 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:51.918 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T03:40:51.962 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:51.963 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-09-22T03:40:51.963 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:52.039 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-fuse 2024-09-22T03:40:52.623 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:14 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:52.756 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:52.756 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:52.756 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-22T03:40:52.756 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:52.757 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:52.758 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 91 k 2024-09-22T03:40:52.758 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 321 k 2024-09-22T03:40:52.758 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:52.957 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 455 kB/s | 91 kB 00:00 2024-09-22T03:40:52.958 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:52.958 INFO:teuthology.orchestra.run.smithi174.stdout:Total 452 kB/s | 91 kB 00:00 2024-09-22T03:40:52.958 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:52.967 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:52.967 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:53.030 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:53.030 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:53.193 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:53.297 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:53.708 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:53.974 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:53.974 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:53.975 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:53.975 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:53.975 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:53.975 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:54.259 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-mirror 2024-09-22T03:40:54.856 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:54.995 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repo Size 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 3.5 M 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 30 M 2024-09-22T03:40:54.996 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:55.343 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 10 MB/s | 3.5 MB 00:00 2024-09-22T03:40:55.344 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:55.344 INFO:teuthology.orchestra.run.smithi174.stdout:Total 10 MB/s | 3.5 MB 00:00 2024-09-22T03:40:55.344 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:55.355 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:55.355 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:55.438 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:55.438 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:56.018 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:56.077 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:56.102 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:56.102 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:40:56.102 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:40:56.102 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:40:56.102 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:40:56.103 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:57.330 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:57.331 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:57.331 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:57.331 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:57.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install rbd-nbd 2024-09-22T03:40:58.193 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:02:20 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:40:58.335 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:40:58.335 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout:Install 1 Package 2024-09-22T03:40:58.336 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:58.337 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 176 k 2024-09-22T03:40:58.337 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 685 k 2024-09-22T03:40:58.337 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:40:58.571 INFO:teuthology.orchestra.run.smithi174.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 754 kB/s | 176 kB 00:00 2024-09-22T03:40:58.572 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:40:58.572 INFO:teuthology.orchestra.run.smithi174.stdout:Total 748 kB/s | 176 kB 00:00 2024-09-22T03:40:58.572 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:40:58.582 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:40:58.582 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:40:58.650 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:40:58.651 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:40:58.823 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:40:58.935 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:59.407 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:40:59.663 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:40:59.934 DEBUG:teuthology.parallel:result is None 2024-09-22T03:40:59.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:41:00.082 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:41:00.103 INFO:teuthology.orchestra.run.smithi174.stdout:19.3.0-5092.g62da54b4.el9 2024-09-22T03:41:00.105 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-22T03:41:00.106 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-22T03:41:00.107 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:41:00.108 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:00.108 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:41:00.169 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:41:00.170 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:00.170 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:41:00.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:41:00.299 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:41:00.299 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:00.299 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:41:00.364 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:41:00.428 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:41:00.428 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:00.428 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:41:00.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:41:00.556 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:41:00.632 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:41:00.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-09-22T03:41:00.657 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:41:00.657 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:41:00.723 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:41:00.723 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:41:00.806 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:41:00.806 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:41:00.806 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914365', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_fio}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914365', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ90l0Km5eIPvnLbI3DjNxDsdOXrpDEa8h1PGIv3rgvP3xGXbjeU6S5YgSgJ/S8FhGIXpwV1HTFATn2l/oqKu9s='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:41:00.807 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:41:00.844 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:41:00.908 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:00.908 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:41:00.964 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:41:00.965 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-22 03:40:56.365149669 +0000 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-22 03:37:51.972012538 +0000 2024-09-22T03:41:01.020 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-22 03:37:51.972012538 +0000 2024-09-22T03:41:01.021 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-22 03:37:51.972012538 +0000 2024-09-22T03:41:01.021 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:41:01.083 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-22T03:41:01.083 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-22T03:41:01.083 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000201892 s, 2.5 MB/s 2024-09-22T03:41:01.084 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:41:01.141 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-22 03:40:56.366149655 +0000 2024-09-22T03:41:01.197 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-22 03:37:52.303008126 +0000 2024-09-22T03:41:01.198 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-22 03:37:52.303008126 +0000 2024-09-22T03:41:01.198 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-22 03:37:52.303008126 +0000 2024-09-22T03:41:01.198 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:41:01.262 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-22T03:41:01.262 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-22T03:41:01.262 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000208053 s, 2.5 MB/s 2024-09-22T03:41:01.263 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:41:01.320 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-22 03:40:56.366149655 +0000 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-22 03:37:52.613003993 +0000 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-22 03:37:52.613003993 +0000 2024-09-22T03:41:01.375 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-22 03:37:52.613003993 +0000 2024-09-22T03:41:01.376 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:41:01.437 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-22T03:41:01.437 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-22T03:41:01.437 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.00016349 s, 3.1 MB/s 2024-09-22T03:41:01.438 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:41:01.495 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-09-22 03:40:56.366149655 +0000 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-09-22 03:37:52.935999687 +0000 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-09-22 03:37:52.935999687 +0000 2024-09-22T03:41:01.551 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-09-22 03:37:52.935999687 +0000 2024-09-22T03:41:01.551 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:41:01.614 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-09-22T03:41:01.614 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-09-22T03:41:01.614 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000217467 s, 2.4 MB/s 2024-09-22T03:41:01.615 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:41:01.672 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:41:01.672 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:41:01.672 INFO:tasks.ceph:Generating config... 2024-09-22T03:41:01.673 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:41:01.673 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:41:01.673 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:41:01.673 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T03:41:01.673 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-22T03:41:01.674 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-22T03:41:01.675 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-22T03:41:01.675 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-22T03:41:01.675 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:41:01.675 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T03:41:01.675 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T03:41:01.676 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T03:41:01.677 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T03:41:01.678 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:41:01.679 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T03:41:01.680 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:41:01.680 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:41:01.680 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:41:01.680 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:41:01.680 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:41:01.680 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:41:01.681 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:41:01.779 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:41:01.782 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:41:01.833 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:41:01.857 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.174')] 2024-09-22T03:41:01.857 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.174', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:41:01.858 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:01.859 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:41:01.913 DEBUG:teuthology.orchestra.run.smithi174:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.174 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:02.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:02.005+0000 7f3ab0e2cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:02.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:02.005+0000 7f3ab0e2cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:02.006 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:02.005+0000 7f3ab0e2cf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:monmaptool: generated fsid 2254448d-7e5e-4d52-88cf-0560050069a8 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:setting min_mon_release = quincy 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:epoch 0 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:fsid 2254448d-7e5e-4d52-88cf-0560050069a8 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:last_changed 2024-09-22T03:41:02.006451+0000 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:created 2024-09-22T03:41:02.006451+0000 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:min_mon_release 17 (quincy) 2024-09-22T03:41:02.007 INFO:teuthology.orchestra.run.smithi174.stdout:election_strategy: 1 2024-09-22T03:41:02.008 INFO:teuthology.orchestra.run.smithi174.stdout:0: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.a 2024-09-22T03:41:02.008 INFO:teuthology.orchestra.run.smithi174.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:41:02.009 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:41:02.024 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2254448d-7e5e-4d52-88cf-0560050069a8... 2024-09-22T03:41:02.025 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = none 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.107 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:41:02.108 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-09-22T03:41:02.109 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = 172.21.15.174 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: auth client required = none 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: auth cluster required = none 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: auth service required = none 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug asok = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug bdev = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug buffer = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug civetweb = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 0/0 2024-09-22T03:41:02.110 INFO:teuthology.orchestra.run.smithi174.stdout: debug compressor = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug context = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug crush = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug crypto = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug dpdk = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug eventtrace = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug filer = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug filestore = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug finisher = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug fuse = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug heartbeatmap = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug javaclient = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug journal = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug journaler = 0/0 2024-09-22T03:41:02.111 INFO:teuthology.orchestra.run.smithi174.stdout: debug kstore = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug leveldb = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug lockdep = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds balancer = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds locker = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds log = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds log expire = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds migrator = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug memdb = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgrc = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 0/0 2024-09-22T03:41:02.112 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug objclass = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug objectcacher = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug objecter = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug optracker = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug perfcounter = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rados = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rbd = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rbd mirror = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rbd replay = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug refs = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rgw = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rgw lifecycle = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rgw notification = 0/0 2024-09-22T03:41:02.113 INFO:teuthology.orchestra.run.smithi174.stdout: debug rgw sync = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: debug striper = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: debug throttle = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: debug timer = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: debug tp = 0/0 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms client mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms cluster mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms mon client mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms mon cluster mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms mon service mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: ms service mode = crc 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 2254448d-7e5e-4d52-88cf-0560050069a8 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.114 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug trim objects = true 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.115 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: debug monc = 20 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:41:02.116 INFO:teuthology.orchestra.run.smithi174.stdout: crimson osd obc lru size = 10 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug filestore = 0/0 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug journal = 0/0 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 0/0 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 0/0 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore backref = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore cache = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore cleaner = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore device = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore epm = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore fixedkv tree = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore journal = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore lba = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore odata = 20 2024-09-22T03:41:02.117 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore ompap = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore onode = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore t = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug seastore tm = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = seastore 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: osd shutdown pgref assert = True 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:41:02.118 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 0/0 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 0/0 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 0/0 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.119 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.120 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default crimson = True 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:41:02.121 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-09-22T03:41:02.122 INFO:teuthology.orchestra.run.smithi174.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:41:02.122 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-09-22T03:41:02.122 INFO:teuthology.orchestra.run.smithi174.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:41:02.123 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:41:02.123 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:41:02.228 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:41:02.228 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:02.228 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:41:02.243 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:02.243 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:41:02.299 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-09-22T03:41:02.299 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:02.299 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:41:02.300 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:41:02.374 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:02.374 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:02.430 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:41:02.430 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:41:02.430 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:41:02.530 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:41:02.533 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:41:02.533 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:41:02.533 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:41:02.582 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:41:02.596 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:41:02.596 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:41:02.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:41:02.620 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:41:02.620 INFO:tasks.ceph:role: osd.0 2024-09-22T03:41:02.620 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-22T03:41:02.620 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:41:02.693 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:41:02.694 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:41:02.694 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:41:02.694 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:41:03.195 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-22T03:41:03.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-22T03:41:03.198 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:41:03.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:41:03.361 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:41:03.427 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:41:03.428 INFO:tasks.ceph:role: osd.1 2024-09-22T03:41:03.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-22T03:41:03.428 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:41:03.501 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:41:03.960 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-22T03:41:03.962 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-22T03:41:03.962 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:41:04.057 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:41:04.087 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:41:04.154 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:41:04.154 INFO:tasks.ceph:role: osd.2 2024-09-22T03:41:04.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-09-22T03:41:04.154 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:41:04.226 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:41:04.226 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:41:04.226 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:41:04.226 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:41:04.227 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:41:04.712 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-09-22T03:41:04.714 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-09-22T03:41:04.714 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:41:04.859 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:41:04.891 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:05.182 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:05.183 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:05,182 seastar - Reactor backend: linux-aio 2024-09-22T03:41:05.202 INFO:teuthology.orchestra.run.smithi174.stderr:==28010==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:05.328 INFO:teuthology.orchestra.run.smithi174.stderr:DEBUG 2024-09-22 03:41:05,328 [shard 0:main] seastar - reactor::drain 2024-09-22T03:41:05.841 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:05.841 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:05.841 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:05.841 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:05.842 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:05.842 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:05.902 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:05.906 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:05,906 seastar - Reactor backend: linux-aio 2024-09-22T03:41:05.915 INFO:teuthology.orchestra.run.smithi174.stderr:==28008==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:06.200 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3767ddb 0xb9ffe85 0xba0b7a4 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.200 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.200 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x97f1b6a 0x97f1d75 0x9826a2f 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.200 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.258 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156f9 0xdd312 0x4ab1c 0x552b064 0x9eb4514 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.258 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4218f 0x3ef7b 0xfd531 0x3c34e46 0x553c478 0x9eb429d 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.258 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.258 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.466 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525e8a 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:06.466 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.523 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd2cf 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:06.523 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.543 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.543 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.546 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525771 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.546 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.601 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.601 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.603 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.603 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.631 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3ad3f 0x10d76c 0xdd31a 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:06.631 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.709 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 230 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10796f 0xdd305 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.709 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:06.908 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x42198 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xba0a005 0xba0a0f4 0xba0a374 0xba0a42d 0xba0a91f 0xba0ade1 0xba0b239 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:06.909 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff883f6041 0xffffffff883f64c0 0xffffffff882908cb 0xffffffff885a8199 0xffffffff8858e203 0xffffffff8859699b 0xffffffff88596e1c 0xffffffff8828c884 0xffffffff88ed1db2 0xffffffff89000bb2 2024-09-22T03:41:07.042 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10798d 0xdd2fc 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:07.042 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:07.099 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16d06 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:07.099 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:07.994 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:07.994 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:07.994 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:07.994 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:07.995 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:07.995 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:08.023 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:41:08.049 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:08.331 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:08.331 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:08,331 seastar - Reactor backend: linux-aio 2024-09-22T03:41:08.337 INFO:teuthology.orchestra.run.smithi174.stderr:==28057==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:08.443 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4abab 0x5fccc96 0x5fcce78 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:08.443 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:08.467 INFO:teuthology.orchestra.run.smithi174.stderr:DEBUG 2024-09-22 03:41:08,467 [shard 0:main] seastar - reactor::drain 2024-09-22T03:41:08.971 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:08.971 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:08.971 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:08.971 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:08.972 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:08.972 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:09.027 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:09.030 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:09,030 seastar - Reactor backend: linux-aio 2024-09-22T03:41:09.040 INFO:teuthology.orchestra.run.smithi174.stderr:==28055==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:09.325 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9ca16f 0xb9f36bb 0xba0b16e 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.325 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.325 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41a36 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xba0a005 0xba0a0f4 0xba0a374 0xba0a91f 0xba0ade1 0xba0b239 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.325 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.382 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e3849e 0x9e7faea 0x9e80601 0x9e80a13 0x9e054ff 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.383 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.383 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x117f8f 0x60cd8e4 0x9eb3dc5 0x9eb457c 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.383 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.383 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x7d4a7 0x5be6a0f 0x5c38d49 0x76be6f4 0xba0e265 0xb9b5385 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:09.383 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.441 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x41fa7 0x3ecea 0xfc9b1 0x3869141 0xb837df3 0xba09156 0xb9ae311 0x9add337 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:09.441 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.573 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x552518c 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:09.573 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.630 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ead 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:09.630 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.663 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab7c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.663 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.721 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab37 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:09.721 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:09.738 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525851 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:09.738 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:10.031 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9cef04 0xb9e7a2b 0xb9e8829 0xba170df 0xba17729 0xb9b1eb4 0x9826b31 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:10.032 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:10.032 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8c12b 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9d02c7 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:10.032 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:10.089 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x162bff 0x422c4 0x3ef7b 0xfd531 0x3c34e46 0x553c478 0x9eb429d 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d9adc 0x92f54b9 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:10.089 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8861d8ac 0xffffffff88550141 0xffffffff88550268 0xffffffff8858ec75 0xffffffff8859699b 0xffffffff88596e1c 0xffffffff8828c884 0xffffffff88ed1db2 0xffffffff89000bb2 2024-09-22T03:41:10.232 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525dcc 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:10.232 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:11.098 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:11.098 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:11.098 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:11.098 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:11.098 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:11.099 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:11.124 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:41:11.148 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:11.436 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:11.436 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:11,436 seastar - Reactor backend: linux-aio 2024-09-22T03:41:11.442 INFO:teuthology.orchestra.run.smithi174.stderr:==28104==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:11.569 INFO:teuthology.orchestra.run.smithi174.stderr:DEBUG 2024-09-22 03:41:11,569 [shard 0:main] seastar - reactor::drain 2024-09-22T03:41:12.070 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:12.071 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:12.071 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:12.071 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:12.071 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:12.071 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:12.132 INFO:teuthology.orchestra.run.smithi174.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:41:12.136 INFO:teuthology.orchestra.run.smithi174.stderr:INFO 2024-09-22 03:41:12,136 seastar - Reactor backend: linux-aio 2024-09-22T03:41:12.156 INFO:teuthology.orchestra.run.smithi174.stderr:==28102==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:41:12.439 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9d06ca 0xba0919c 0xb9ae311 0x9add337 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.439 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c2b 0xdd332 0x4ab1c 0x5c73334 0xb9cdb42 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.439 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.439 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.496 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55252e1 0x552b009 0x9eb4514 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.496 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x117f8f 0x60cd8e4 0x9eb3dc5 0x9eb457c 0x9eb4856 0x9eb4bfe 0x9eb7fd8 0x9eb8c34 0x9eb8ef7 0x9eb9b6e 0x9e050d2 0x99e3351 0x99e3957 0x98d8890 0x9311591 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.496 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.496 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.497 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4bfbf 0xb9e0c3c 0xba1c4c3 0xba1c84e 0xb9bcc48 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:12.497 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.555 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797f 0xdd2fc 0x4b14c 0xba0adf0 0xba0b239 0xb9aee06 0xba0b778 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:12.555 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.686 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55257e2 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:12.687 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.744 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525380 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:12.744 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.796 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c3f 0xdd332 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.796 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.796 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ea2 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.796 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.851 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 230 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525ea2 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T03:41:12.852 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.853 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525379 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.853 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:12.853 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 122 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525cbb 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:12.853 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:13.168 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4b12f 0x60f7453 0x60f75d7 0x60f785f 0xba17baf 0xba17ce0 0xba17ecc 0xba18068 0xba183fe 0xba18610 0xb9b548f 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:13.168 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:13.168 INFO:teuthology.orchestra.run.smithi174.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10798a 0xdd2e1 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x92f52b6 0x92f5d5a 0x92f5ddd 0x92f5e62 0x92f5f64 0x92f605c 0x92f60e2 0x92f619b 0x92f6351 0x92f6489 0x92f6504 0x92f6589 0x92f66e4 0x92f67fe 0x92f6883 0x92f6961 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T03:41:13.168 INFO:teuthology.orchestra.run.smithi174.stderr:kernel callstack: 2024-09-22T03:41:14.290 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:14.290 INFO:teuthology.orchestra.run.smithi174.stderr:Suppressions used: 2024-09-22T03:41:14.290 INFO:teuthology.orchestra.run.smithi174.stderr: count bytes template 2024-09-22T03:41:14.290 INFO:teuthology.orchestra.run.smithi174.stderr: 1 8 MallocExtension::Register 2024-09-22T03:41:14.290 INFO:teuthology.orchestra.run.smithi174.stderr:----------------------------------------------------- 2024-09-22T03:41:14.291 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:14.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:41:14.342 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:41:14.342 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:14.343 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:41:14.407 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:14.407 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:41:14.471 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:14.471 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:41:14.534 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:14.535 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:41:14.599 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:14.599 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:41:14.654 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:41:14.654 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:41:14.716 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-09-22T03:41:14.716 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDOke9mOCeYHxAABvQdHkBqKKmK/i+iKLWsVw== 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDSke9mNceAAxAAxIFXWVrm6NbseX8SLLNNGQ== 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDVke9mlOf6ChAA2/ofZK4dXR/S3soSKgRU+w== 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDYke9mP9PnERAAIG0bDoRUplrWbQ50A6a8Ww== 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-09-22T03:41:14.717 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQDOke9meGC6IhAAPVVASkQLQj0CP31ZtjexiA== 2024-09-22T03:41:14.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:41:14.819 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:41:14.880 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:41:14.940 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:41:15.001 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:41:15.065 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:41:15.065 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:41:15.089 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:41:15.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.198+0000 7f988e8fbc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.198+0000 7f988e8fbc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.214 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:41:15.239 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:41:15.295 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:41:15.295 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:41:15.295 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:41:15.337 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:41:15.337 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:41:15.338 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:41:15.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:41:15.340 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:41:15.340 DEBUG:tasks.ceph:set 0 configs 2024-09-22T03:41:15.340 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:41:15.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:41:15.421 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:41:15.419+0000 7feb07e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.421 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:41:15.419+0000 7feb07e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.422 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:41:15.419+0000 7fa1049e3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.422 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:41:15.419+0000 7feb07e2fc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.442+0000 7f47e86fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.452 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.450+0000 7f47e86fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.519 INFO:teuthology.orchestra.run.smithi174.stderr:adjusted tunables profile to default 2024-09-22T03:41:15.531 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:41:15.531 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:41:15.531 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:41:15.625 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:41:15.623+0000 7fa1049e3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.625 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:41:15.623+0000 7fa1049e3000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.635 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.629+0000 7f7563751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.638 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:15.636+0000 7f7563751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:41:15.715+0000 7feafe7f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr:feature with important data. 2024-09-22T03:41:15.717 INFO:tasks.ceph.mon.a.smithi174.stderr: 2024-09-22T03:41:16.532 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:41:16.532 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:16.533 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:41:16.557 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:16.558 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:41:16.622 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:16.622 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:41:16.688 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new fc6816ac-885e-4678-82b3-6cb214ac7492 0 2024-09-22T03:41:16.836 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:16.828+0000 7fd9caaac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:16.838 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:16.836+0000 7fd9caaac640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:16.917 INFO:teuthology.orchestra.run.smithi174.stdout:0 2024-09-22T03:41:16.929 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 008acea5-78d3-467b-a933-c0120a0e0e55 1 2024-09-22T03:41:17.038 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.031+0000 7f2d21991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.040 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.038+0000 7f2d21991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.117 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-09-22T03:41:17.129 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 971ad6b9-05d1-49ac-8dd9-fb43dae70f7a 2 2024-09-22T03:41:17.238 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.230+0000 7ffb9c591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.240 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.238+0000 7ffb9c591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.321 INFO:teuthology.orchestra.run.smithi174.stdout:2 2024-09-22T03:41:17.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:41:17.362 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:41:17.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:41:17.404 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:41:17.404 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:41:17.432 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:41:17.432 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:41:17.475 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:41:17.475 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:41:17.507 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:41:17.507 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:41:17.549 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:41:17.550 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:41:17.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.718+0000 7f92b2991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.727 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:17.725+0000 7f92b2991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:17.795 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:17.795 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":6,"fsid":"2254448d-7e5e-4d52-88cf-0560050069a8","created":"2024-09-22T03:41:15.444248+0000","modified":"2024-09-22T03:41:17.319959+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:41:17.319959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"fc6816ac-885e-4678-82b3-6cb214ac7492","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"008acea5-78d3-467b-a933-c0120a0e0e55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"971ad6b9-05d1-49ac-8dd9-fb43dae70f7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:41:17.807 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:41:17.807 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:41:18.109 DEBUG:teuthology.orchestra.run.smithi174:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:41:18.214 INFO:teuthology.misc.health.smithi174.stderr:2024-09-22T03:41:18.208+0000 7f5d11791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:18.216 INFO:teuthology.misc.health.smithi174.stderr:2024-09-22T03:41:18.214+0000 7f5d11791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:18.284 INFO:teuthology.misc.health.smithi174.stdout: 2024-09-22T03:41:18.284 INFO:teuthology.misc.health.smithi174.stdout:{"epoch":6,"fsid":"2254448d-7e5e-4d52-88cf-0560050069a8","created":"2024-09-22T03:41:15.444248+0000","modified":"2024-09-22T03:41:17.319959+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:41:17.319959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"fc6816ac-885e-4678-82b3-6cb214ac7492","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"008acea5-78d3-467b-a933-c0120a0e0e55","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"971ad6b9-05d1-49ac-8dd9-fb43dae70f7a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:41:18.296 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:41:21.003 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:41:21.001+0000 7fa0a1c6d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:41:24.599 DEBUG:teuthology.orchestra.run.smithi174:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:41:24.716 INFO:teuthology.misc.health.smithi174.stderr:2024-09-22T03:41:24.710+0000 7f9f162d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:24.719 INFO:teuthology.misc.health.smithi174.stderr:2024-09-22T03:41:24.717+0000 7f9f162d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:24.918 INFO:teuthology.misc.health.smithi174.stdout: 2024-09-22T03:41:24.918 INFO:teuthology.misc.health.smithi174.stdout:{"epoch":10,"fsid":"2254448d-7e5e-4d52-88cf-0560050069a8","created":"2024-09-22T03:41:15.444248+0000","modified":"2024-09-22T03:41:24.060243+0000","last_up_change":"2024-09-22T03:41:24.060243+0000","last_in_change":"2024-09-22T03:41:17.319959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"fc6816ac-885e-4678-82b3-6cb214ac7492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":898621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":898621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6815","nonce":898621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":898621689}]},"public_addr":"172.21.15.174:6807/898621689","cluster_addr":"172.21.15.174:6806/898621689","heartbeat_back_addr":"172.21.15.174:6815/898621689","heartbeat_front_addr":"172.21.15.174:6814/898621689","state":["exists","up"]},{"osd":1,"uuid":"008acea5-78d3-467b-a933-c0120a0e0e55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":3033857583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3033857583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":3033857583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3033857583}]},"public_addr":"172.21.15.174:6805/3033857583","cluster_addr":"172.21.15.174:6804/3033857583","heartbeat_back_addr":"172.21.15.174:6803/3033857583","heartbeat_front_addr":"172.21.15.174:6802/3033857583","state":["exists","up"]},{"osd":2,"uuid":"971ad6b9-05d1-49ac-8dd9-fb43dae70f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":2211395004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2211395004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2211395004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":2211395004}]},"public_addr":"172.21.15.174:6809/2211395004","cluster_addr":"172.21.15.174:6808/2211395004","heartbeat_back_addr":"172.21.15.174:6810/2211395004","heartbeat_front_addr":"172.21.15.174:6811/2211395004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:41:24.933 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:41:24.933 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:41:24.933 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:41:25.053 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:25.043+0000 7f481b311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:25.055 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:25.054+0000 7f481b311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:26.071 INFO:teuthology.orchestra.run.smithi174.stderr:pool 'rbd' created 2024-09-22T03:41:26.098 DEBUG:teuthology.orchestra.run.smithi174:> rbd --cluster ceph pool init rbd 2024-09-22T03:41:26.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:26.213+0000 7f702e60eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:26.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:26.213+0000 7f702e60eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:26.218 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:26.216+0000 7f702e60eac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.152 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:41:29.152 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-22T03:41:29.153 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:41:29.264 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.258+0000 7fed57ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.266 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.264+0000 7fed57ed1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.450 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:29.463 INFO:teuthology.orchestra.run.smithi174.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-22T03:41:29.464 INFO:tasks.ceph_manager:config epoch is 1 2024-09-22T03:41:29.464 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:41:29.464 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:41:29.464 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:41:29.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.569+0000 7f8038311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.573+0000 7f8038311640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.808 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:29.820 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":2021771961},{"type":"v1","addr":"172.21.15.174:6801","nonce":2021771961}]},"active_addr":"172.21.15.174:6801/2021771961","active_change":"2024-09-22T03:41:19.986334+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":2123417475}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":3124609550}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":599624957}]}]} 2024-09-22T03:41:29.823 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:41:29.823 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:41:29.823 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:41:29.932 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.928+0000 7f82d34d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:29.934 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:29.932+0000 7f82d34d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.123 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:30.123 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":14,"fsid":"2254448d-7e5e-4d52-88cf-0560050069a8","created":"2024-09-22T03:41:15.444248+0000","modified":"2024-09-22T03:41:29.077979+0000","last_up_change":"2024-09-22T03:41:24.060243+0000","last_in_change":"2024-09-22T03:41:17.319959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:41:25.270772+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:41:26.005944+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fc6816ac-885e-4678-82b3-6cb214ac7492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":898621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":898621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6815","nonce":898621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":898621689}]},"public_addr":"172.21.15.174:6807/898621689","cluster_addr":"172.21.15.174:6806/898621689","heartbeat_back_addr":"172.21.15.174:6815/898621689","heartbeat_front_addr":"172.21.15.174:6814/898621689","state":["exists","up"]},{"osd":1,"uuid":"008acea5-78d3-467b-a933-c0120a0e0e55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":3033857583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3033857583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":3033857583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3033857583}]},"public_addr":"172.21.15.174:6805/3033857583","cluster_addr":"172.21.15.174:6804/3033857583","heartbeat_back_addr":"172.21.15.174:6803/3033857583","heartbeat_front_addr":"172.21.15.174:6802/3033857583","state":["exists","up"]},{"osd":2,"uuid":"971ad6b9-05d1-49ac-8dd9-fb43dae70f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":2211395004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2211395004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2211395004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":2211395004}]},"public_addr":"172.21.15.174:6809/2211395004","cluster_addr":"172.21.15.174:6808/2211395004","heartbeat_back_addr":"172.21.15.174:6810/2211395004","heartbeat_front_addr":"172.21.15.174:6811/2211395004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:41:30.136 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:41:30.136 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:41:30.252 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.248+0000 7feed60fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.254 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.252+0000 7feed60fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.441 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:30.441 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":14,"fsid":"2254448d-7e5e-4d52-88cf-0560050069a8","created":"2024-09-22T03:41:15.444248+0000","modified":"2024-09-22T03:41:29.077979+0000","last_up_change":"2024-09-22T03:41:24.060243+0000","last_in_change":"2024-09-22T03:41:17.319959+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T03:41:25.270772+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T03:41:26.005944+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fc6816ac-885e-4678-82b3-6cb214ac7492","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6807","nonce":898621689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":898621689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6815","nonce":898621689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":898621689}]},"public_addr":"172.21.15.174:6807/898621689","cluster_addr":"172.21.15.174:6806/898621689","heartbeat_back_addr":"172.21.15.174:6815/898621689","heartbeat_front_addr":"172.21.15.174:6814/898621689","state":["exists","up"]},{"osd":1,"uuid":"008acea5-78d3-467b-a933-c0120a0e0e55","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6805","nonce":3033857583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3033857583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6803","nonce":3033857583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3033857583}]},"public_addr":"172.21.15.174:6805/3033857583","cluster_addr":"172.21.15.174:6804/3033857583","heartbeat_back_addr":"172.21.15.174:6803/3033857583","heartbeat_front_addr":"172.21.15.174:6802/3033857583","state":["exists","up"]},{"osd":2,"uuid":"971ad6b9-05d1-49ac-8dd9-fb43dae70f7a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6809","nonce":2211395004}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2211395004}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":2211395004}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6811","nonce":2211395004}]},"public_addr":"172.21.15.174:6809/2211395004","cluster_addr":"172.21.15.174:6808/2211395004","heartbeat_back_addr":"172.21.15.174:6810/2211395004","heartbeat_front_addr":"172.21.15.174:6811/2211395004","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:41:30.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:41:30.455 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:41:30.456 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:41:30.570 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.564+0000 7f521c2bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.564+0000 7f2034991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.564+0000 7f8827d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.571+0000 7f521c2bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.572+0000 7f8827d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.573+0000 7f2034991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.600 INFO:teuthology.orchestra.run.smithi174.stdout:42949672966 2024-09-22T03:41:30.601 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:41:30.605 INFO:teuthology.orchestra.run.smithi174.stdout:34359738376 2024-09-22T03:41:30.605 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:41:30.606 INFO:teuthology.orchestra.run.smithi174.stdout:42949672966 2024-09-22T03:41:30.606 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:41:30.713 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.707+0000 7f3e2fad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.716 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.714+0000 7f3e2fad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.759 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.753+0000 7f3b41991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.760 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.753+0000 7f0b22791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.761 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.759+0000 7f3b41991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.762 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:30.760+0000 7f0b22791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:30.915 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:30.928 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:41:30.961 INFO:teuthology.orchestra.run.smithi174.stdout:34359738373 2024-09-22T03:41:30.967 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:30.975 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-22T03:41:30.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:41:31.929 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:41:31.976 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:41:31.981 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:41:32.056 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.050+0000 7faba02fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.060 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.059+0000 7faba02fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.099 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.092+0000 7f4624b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.101 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.100+0000 7f4624b91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.136 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.131+0000 7f0f3a791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.138 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:32.137+0000 7f0f3a791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:32.263 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:32.276 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:41:32.303 INFO:teuthology.orchestra.run.smithi174.stdout:34359738373 2024-09-22T03:41:32.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-22T03:41:32.337 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:32.350 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:41:33.277 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:41:33.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:41:33.350 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:41:33.395 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.390+0000 7fa175d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.398 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.397+0000 7fa175d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.431+0000 7feba6a7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.439 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.438+0000 7feba6a7d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.480 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.475+0000 7f59edd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.482 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:33.481+0000 7f59edd91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:33.625 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:33.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:41:33.664 INFO:teuthology.orchestra.run.smithi174.stdout:34359738373 2024-09-22T03:41:33.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738373 for osd.1 2024-09-22T03:41:33.680 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:33.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:41:34.644 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:41:34.679 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:41:34.693 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:41:34.760 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.754+0000 7fdb2a351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.762 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.761+0000 7fdb2a351640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.802 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.796+0000 7ffa6d591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.804+0000 7ffa6d591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.844 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.839+0000 7f73e1191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.846 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:34.846+0000 7f73e1191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:34.971 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:34.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T03:41:35.002 INFO:teuthology.orchestra.run.smithi174.stdout:34359738378 2024-09-22T03:41:35.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738378 for osd.1 2024-09-22T03:41:35.016 DEBUG:teuthology.parallel:result is None 2024-09-22T03:41:35.052 INFO:teuthology.orchestra.run.smithi174.stdout:42949672964 2024-09-22T03:41:35.065 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T03:41:35.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:41:36.066 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:41:36.102 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.097+0000 7f7094391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.104 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.104+0000 7f7094391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.180 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.174+0000 7f238fa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.182 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.181+0000 7f238fa6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.298 INFO:teuthology.orchestra.run.smithi174.stdout:42949672969 2024-09-22T03:41:36.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.2 2024-09-22T03:41:36.311 DEBUG:teuthology.parallel:result is None 2024-09-22T03:41:36.376 INFO:teuthology.orchestra.run.smithi174.stdout:42949672969 2024-09-22T03:41:36.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.0 2024-09-22T03:41:36.389 DEBUG:teuthology.parallel:result is None 2024-09-22T03:41:36.389 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:41:36.390 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:41:36.498 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.493+0000 7f7c4ccbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.500 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.499+0000 7f7c4ccbd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.684 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:36.684 INFO:teuthology.orchestra.run.smithi174.stderr:dumped all 2024-09-22T03:41:36.696 INFO:teuthology.orchestra.run.smithi174.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T03:41:35.992149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.903420"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110626+0000","last_change":"2024-09-22T03:41:29.116927+0000","last_active":"2024-09-22T03:41:29.110626+0000","last_peered":"2024-09-22T03:41:29.110626+0000","last_clean":"2024-09-22T03:41:29.110626+0000","last_became_active":"2024-09-22T03:41:27.268321+0000","last_became_peered":"2024-09-22T03:41:27.268321+0000","last_unstale":"2024-09-22T03:41:29.110626+0000","last_undegraded":"2024-09-22T03:41:29.110626+0000","last_fullsized":"2024-09-22T03:41:29.110626+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.106913+0000","last_change":"2024-09-22T03:41:29.112229+0000","last_active":"2024-09-22T03:41:29.106913+0000","last_peered":"2024-09-22T03:41:29.106913+0000","last_clean":"2024-09-22T03:41:29.106913+0000","last_became_active":"2024-09-22T03:41:27.278776+0000","last_became_peered":"2024-09-22T03:41:27.278776+0000","last_unstale":"2024-09-22T03:41:29.106913+0000","last_undegraded":"2024-09-22T03:41:29.106913+0000","last_fullsized":"2024-09-22T03:41:29.106913+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.139718+0000","last_change":"2024-09-22T03:41:29.142361+0000","last_active":"2024-09-22T03:41:29.139718+0000","last_peered":"2024-09-22T03:41:29.139718+0000","last_clean":"2024-09-22T03:41:29.139718+0000","last_became_active":"2024-09-22T03:41:27.254588+0000","last_became_peered":"2024-09-22T03:41:27.254588+0000","last_unstale":"2024-09-22T03:41:29.139718+0000","last_undegraded":"2024-09-22T03:41:29.139718+0000","last_fullsized":"2024-09-22T03:41:29.139718+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110197+0000","last_change":"2024-09-22T03:41:29.112603+0000","last_active":"2024-09-22T03:41:29.110197+0000","last_peered":"2024-09-22T03:41:29.110197+0000","last_clean":"2024-09-22T03:41:29.110197+0000","last_became_active":"2024-09-22T03:41:27.214158+0000","last_became_peered":"2024-09-22T03:41:27.214158+0000","last_unstale":"2024-09-22T03:41:29.110197+0000","last_undegraded":"2024-09-22T03:41:29.110197+0000","last_fullsized":"2024-09-22T03:41:29.110197+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.142861+0000","last_change":"2024-09-22T03:41:27.263290+0000","last_active":"2024-09-22T03:41:29.142861+0000","last_peered":"2024-09-22T03:41:29.142861+0000","last_clean":"2024-09-22T03:41:29.142861+0000","last_became_active":"2024-09-22T03:41:27.209510+0000","last_became_peered":"2024-09-22T03:41:27.209510+0000","last_unstale":"2024-09-22T03:41:29.142861+0000","last_undegraded":"2024-09-22T03:41:29.142861+0000","last_fullsized":"2024-09-22T03:41:29.142861+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110427+0000","last_change":"2024-09-22T03:41:29.117096+0000","last_active":"2024-09-22T03:41:29.110427+0000","last_peered":"2024-09-22T03:41:29.110427+0000","last_clean":"2024-09-22T03:41:29.110427+0000","last_became_active":"2024-09-22T03:41:27.314704+0000","last_became_peered":"2024-09-22T03:41:27.314704+0000","last_unstale":"2024-09-22T03:41:29.110427+0000","last_undegraded":"2024-09-22T03:41:29.110427+0000","last_fullsized":"2024-09-22T03:41:29.110427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.122738+0000","last_change":"2024-09-22T03:41:29.122738+0000","last_active":"2024-09-22T03:41:29.122738+0000","last_peered":"2024-09-22T03:41:29.122738+0000","last_clean":"2024-09-22T03:41:29.122738+0000","last_became_active":"2024-09-22T03:41:27.191925+0000","last_became_peered":"2024-09-22T03:41:27.191925+0000","last_unstale":"2024-09-22T03:41:29.122738+0000","last_undegraded":"2024-09-22T03:41:29.122738+0000","last_fullsized":"2024-09-22T03:41:29.122738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.143011+0000","last_change":"2024-09-22T03:41:29.145430+0000","last_active":"2024-09-22T03:41:29.143011+0000","last_peered":"2024-09-22T03:41:29.143011+0000","last_clean":"2024-09-22T03:41:29.143011+0000","last_became_active":"2024-09-22T03:41:27.361798+0000","last_became_peered":"2024-09-22T03:41:27.361798+0000","last_unstale":"2024-09-22T03:41:29.143011+0000","last_undegraded":"2024-09-22T03:41:29.143011+0000","last_fullsized":"2024-09-22T03:41:29.143011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.108200+0000","last_change":"2024-09-22T03:41:29.112304+0000","last_active":"2024-09-22T03:41:29.108200+0000","last_peered":"2024-09-22T03:41:29.108200+0000","last_clean":"2024-09-22T03:41:29.108200+0000","last_became_active":"2024-09-22T03:41:27.275400+0000","last_became_peered":"2024-09-22T03:41:27.275400+0000","last_unstale":"2024-09-22T03:41:29.108200+0000","last_undegraded":"2024-09-22T03:41:29.108200+0000","last_fullsized":"2024-09-22T03:41:29.108200+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:41:36.697 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:41:36.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.802+0000 7f51a7951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.808 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:36.807+0000 7f51a7951640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:36.991 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:36.991 INFO:teuthology.orchestra.run.smithi174.stderr:dumped all 2024-09-22T03:41:37.003 INFO:teuthology.orchestra.run.smithi174.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T03:41:35.992149+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590387,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.903420"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110626+0000","last_change":"2024-09-22T03:41:29.116927+0000","last_active":"2024-09-22T03:41:29.110626+0000","last_peered":"2024-09-22T03:41:29.110626+0000","last_clean":"2024-09-22T03:41:29.110626+0000","last_became_active":"2024-09-22T03:41:27.268321+0000","last_became_peered":"2024-09-22T03:41:27.268321+0000","last_unstale":"2024-09-22T03:41:29.110626+0000","last_undegraded":"2024-09-22T03:41:29.110626+0000","last_fullsized":"2024-09-22T03:41:29.110626+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.106913+0000","last_change":"2024-09-22T03:41:29.112229+0000","last_active":"2024-09-22T03:41:29.106913+0000","last_peered":"2024-09-22T03:41:29.106913+0000","last_clean":"2024-09-22T03:41:29.106913+0000","last_became_active":"2024-09-22T03:41:27.278776+0000","last_became_peered":"2024-09-22T03:41:27.278776+0000","last_unstale":"2024-09-22T03:41:29.106913+0000","last_undegraded":"2024-09-22T03:41:29.106913+0000","last_fullsized":"2024-09-22T03:41:29.106913+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.139718+0000","last_change":"2024-09-22T03:41:29.142361+0000","last_active":"2024-09-22T03:41:29.139718+0000","last_peered":"2024-09-22T03:41:29.139718+0000","last_clean":"2024-09-22T03:41:29.139718+0000","last_became_active":"2024-09-22T03:41:27.254588+0000","last_became_peered":"2024-09-22T03:41:27.254588+0000","last_unstale":"2024-09-22T03:41:29.139718+0000","last_undegraded":"2024-09-22T03:41:29.139718+0000","last_fullsized":"2024-09-22T03:41:29.139718+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110197+0000","last_change":"2024-09-22T03:41:29.112603+0000","last_active":"2024-09-22T03:41:29.110197+0000","last_peered":"2024-09-22T03:41:29.110197+0000","last_clean":"2024-09-22T03:41:29.110197+0000","last_became_active":"2024-09-22T03:41:27.214158+0000","last_became_peered":"2024-09-22T03:41:27.214158+0000","last_unstale":"2024-09-22T03:41:29.110197+0000","last_undegraded":"2024-09-22T03:41:29.110197+0000","last_fullsized":"2024-09-22T03:41:29.110197+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.142861+0000","last_change":"2024-09-22T03:41:27.263290+0000","last_active":"2024-09-22T03:41:29.142861+0000","last_peered":"2024-09-22T03:41:29.142861+0000","last_clean":"2024-09-22T03:41:29.142861+0000","last_became_active":"2024-09-22T03:41:27.209510+0000","last_became_peered":"2024-09-22T03:41:27.209510+0000","last_unstale":"2024-09-22T03:41:29.142861+0000","last_undegraded":"2024-09-22T03:41:29.142861+0000","last_fullsized":"2024-09-22T03:41:29.142861+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.110427+0000","last_change":"2024-09-22T03:41:29.117096+0000","last_active":"2024-09-22T03:41:29.110427+0000","last_peered":"2024-09-22T03:41:29.110427+0000","last_clean":"2024-09-22T03:41:29.110427+0000","last_became_active":"2024-09-22T03:41:27.314704+0000","last_became_peered":"2024-09-22T03:41:27.314704+0000","last_unstale":"2024-09-22T03:41:29.110427+0000","last_undegraded":"2024-09-22T03:41:29.110427+0000","last_fullsized":"2024-09-22T03:41:29.110427+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.122738+0000","last_change":"2024-09-22T03:41:29.122738+0000","last_active":"2024-09-22T03:41:29.122738+0000","last_peered":"2024-09-22T03:41:29.122738+0000","last_clean":"2024-09-22T03:41:29.122738+0000","last_became_active":"2024-09-22T03:41:27.191925+0000","last_became_peered":"2024-09-22T03:41:27.191925+0000","last_unstale":"2024-09-22T03:41:29.122738+0000","last_undegraded":"2024-09-22T03:41:29.122738+0000","last_fullsized":"2024-09-22T03:41:29.122738+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.143011+0000","last_change":"2024-09-22T03:41:29.145430+0000","last_active":"2024-09-22T03:41:29.143011+0000","last_peered":"2024-09-22T03:41:29.143011+0000","last_clean":"2024-09-22T03:41:29.143011+0000","last_became_active":"2024-09-22T03:41:27.361798+0000","last_became_peered":"2024-09-22T03:41:27.361798+0000","last_unstale":"2024-09-22T03:41:29.143011+0000","last_undegraded":"2024-09-22T03:41:29.143011+0000","last_fullsized":"2024-09-22T03:41:29.143011+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T03:41:29.108200+0000","last_change":"2024-09-22T03:41:29.112304+0000","last_active":"2024-09-22T03:41:29.108200+0000","last_peered":"2024-09-22T03:41:29.108200+0000","last_clean":"2024-09-22T03:41:29.108200+0000","last_became_active":"2024-09-22T03:41:27.275400+0000","last_became_peered":"2024-09-22T03:41:27.275400+0000","last_unstale":"2024-09-22T03:41:29.108200+0000","last_undegraded":"2024-09-22T03:41:29.108200+0000","last_fullsized":"2024-09-22T03:41:29.108200+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:41:26.066625+0000","last_clean_scrub_stamp":"2024-09-22T03:41:26.066625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:41:37.004 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:41:37.004 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:41:37.004 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:41:37.004 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:41:37.110 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:37.107+0000 7fb0ef391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:37.112 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22T03:41:37.111+0000 7fb0ef391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:41:37.337 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:37.337 INFO:teuthology.orchestra.run.smithi174.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:41:37.350 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:41:37.350 INFO:teuthology.run_tasks:Running task cbt... 2024-09-22T03:41:37.364 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi174.front.sepia.ceph.com', 'osds': ['ubuntu@smithi174.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi174.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi174.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi174.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-22T03:41:37.364 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-22T03:41:37.381 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-09-22T03:41:37.382 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:41:37.435 DEBUG:teuthology.orchestra.run.smithi174:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-22T03:41:37.490 INFO:teuthology.orchestra.run.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-22T03:41:37.788 INFO:tasks.cbt:Installing collectl 2024-09-22T03:41:37.788 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-22T03:41:37.882 INFO:teuthology.orchestra.run.smithi174.stderr:--2024-09-22 03:41:37-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-22T03:41:38.028 INFO:teuthology.orchestra.run.smithi174.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-09-22T03:41:38.036 INFO:teuthology.orchestra.run.smithi174.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-09-22T03:41:38.323 INFO:teuthology.orchestra.run.smithi174.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:41:38.323 INFO:teuthology.orchestra.run.smithi174.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75Hylb_vqo1YD7yUb1Zjx-XqFrVrzesXScQr7psE1OHwUB_CuNupNZlj97w_o_gROUl44GPekNeHU9BfAwy5okir5Q%3D%3D&use_mirror=netactuate&r= [following] 2024-09-22T03:41:38.323 INFO:teuthology.orchestra.run.smithi174.stderr:--2024-09-22 03:41:38-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75Hylb_vqo1YD7yUb1Zjx-XqFrVrzesXScQr7psE1OHwUB_CuNupNZlj97w_o_gROUl44GPekNeHU9BfAwy5okir5Q%3D%3D&use_mirror=netactuate&r= 2024-09-22T03:41:38.324 INFO:teuthology.orchestra.run.smithi174.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-22T03:41:38.394 INFO:teuthology.orchestra.run.smithi174.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-22T03:41:38.673 INFO:teuthology.orchestra.run.smithi174.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:41:38.673 INFO:teuthology.orchestra.run.smithi174.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-22T03:41:38.673 INFO:teuthology.orchestra.run.smithi174.stderr:--2024-09-22 03:41:38-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-22T03:41:38.684 INFO:teuthology.orchestra.run.smithi174.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2024-09-22T03:41:38.705 INFO:teuthology.orchestra.run.smithi174.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2024-09-22T03:41:38.855 INFO:teuthology.orchestra.run.smithi174.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-22T03:41:38.855 INFO:teuthology.orchestra.run.smithi174.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-22T03:41:38.855 INFO:teuthology.orchestra.run.smithi174.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-22T03:41:38.855 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:38.875 INFO:teuthology.orchestra.run.smithi174.stderr: 0K .......... .......... .......... .......... .......... 8% 2.40M 0s 2024-09-22T03:41:38.896 INFO:teuthology.orchestra.run.smithi174.stderr: 50K .......... .......... .......... .......... .......... 16% 2.38M 0s 2024-09-22T03:41:38.917 INFO:teuthology.orchestra.run.smithi174.stderr: 100K .......... .......... .......... .......... .......... 24% 2.36M 0s 2024-09-22T03:41:38.917 INFO:teuthology.orchestra.run.smithi174.stderr: 150K .......... .......... .......... .......... .......... 32% 233M 0s 2024-09-22T03:41:38.939 INFO:teuthology.orchestra.run.smithi174.stderr: 200K .......... .......... .......... .......... .......... 41% 2.46M 0s 2024-09-22T03:41:38.939 INFO:teuthology.orchestra.run.smithi174.stderr: 250K .......... .......... .......... .......... .......... 49% 158M 0s 2024-09-22T03:41:38.939 INFO:teuthology.orchestra.run.smithi174.stderr: 300K .......... .......... .......... .......... .......... 57% 113M 0s 2024-09-22T03:41:38.940 INFO:teuthology.orchestra.run.smithi174.stderr: 350K .......... .......... .......... .......... .......... 65% 93.6M 0s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 400K .......... .......... .......... .......... .......... 74% 2.59M 0s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 450K .......... .......... .......... .......... .......... 82% 116M 0s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 500K .......... .......... .......... .......... .......... 90% 119M 0s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 550K .......... .......... .......... .......... .......... 98% 116M 0s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 600K ...... 100% 11.4T=0.1s 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr:2024-09-22 03:41:38 (5.75 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-22T03:41:38.959 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-09-22T03:41:38.984 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/ 2024-09-22T03:41:38.984 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/collectl.conf 2024-09-22T03:41:38.991 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/ 2024-09-22T03:41:38.991 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-22T03:41:38.991 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Socket.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/colmux.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Import.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/style.css 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Process.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-22T03:41:38.992 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Export.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Startup.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Performance.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Messages.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Logging.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-22T03:41:38.993 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Misc.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Examples.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Data.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-22T03:41:38.994 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Playback.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Hello.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Network.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Memory.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/Disks.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/service/ 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/service/collectl.service 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/initd/ 2024-09-22T03:41:38.995 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/initd/collectl 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/misc.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/gexpr.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/statsd.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/envrules.std 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/hello.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/vmsum.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/proctree.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/README-WINDOWS 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/formatit.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/colmux 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/INSTALL 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/graphite.ph 2024-09-22T03:41:38.996 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/vmstat.ph 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/README 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/man1/ 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/man1/collectl.1 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/man1/colmux.1 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/lexpr.ph 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/client.pl 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/collectl 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/COPYING 2024-09-22T03:41:38.998 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/vnet.ph 2024-09-22T03:41:38.999 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/GPL 2024-09-22T03:41:38.999 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/UNINSTALL 2024-09-22T03:41:38.999 INFO:teuthology.orchestra.run.smithi174.stdout:collectl-4.3.1/ARTISTIC 2024-09-22T03:41:39.046 INFO:teuthology.orchestra.run.smithi174.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-22T03:41:39.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-22T03:41:39.637 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:03:01 ago on Sun 22 Sep 2024 03:38:38 AM UTC. 2024-09-22T03:41:39.729 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-22T03:41:39.731 INFO:teuthology.orchestra.run.smithi174.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-22T03:41:39.779 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-22T03:41:39.780 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout:Install 5 Packages 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:39.781 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 2.4 M 2024-09-22T03:41:39.782 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 11 M 2024-09-22T03:41:39.782 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-09-22T03:41:40.182 INFO:teuthology.orchestra.run.smithi174.stdout:(1/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 158 kB/s | 24 kB 00:00 2024-09-22T03:41:40.207 INFO:teuthology.orchestra.run.smithi174.stdout:(2/5): libradospp-devel-19.3.0-5092.g62da54b4.e 175 kB/s | 31 kB 00:00 2024-09-22T03:41:40.257 INFO:teuthology.orchestra.run.smithi174.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 262 kB/s | 13 kB 00:00 2024-09-22T03:41:40.283 INFO:teuthology.orchestra.run.smithi174.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 1.2 MB/s | 118 kB 00:00 2024-09-22T03:41:41.066 INFO:teuthology.orchestra.run.smithi174.stdout:(5/5): perf-5.14.0-508.el9.x86_64.rpm 2.1 MB/s | 2.2 MB 00:01 2024-09-22T03:41:41.069 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:41:41.070 INFO:teuthology.orchestra.run.smithi174.stdout:Total 1.9 MB/s | 2.4 MB 00:01 2024-09-22T03:41:41.099 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-09-22T03:41:41.120 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-09-22T03:41:41.120 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-09-22T03:41:41.253 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-09-22T03:41:41.254 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-09-22T03:41:41.619 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-09-22T03:41:41.701 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-22T03:41:41.731 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-22T03:41:41.754 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-22T03:41:42.081 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-22T03:41:42.159 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:41:43.848 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:41:43.848 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-22T03:41:43.848 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-22T03:41:43.848 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-22T03:41:43.848 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-22T03:41:44.173 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: pdsh-2.34-7.el9.x86_64 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: perf-5.14.0-508.el9.x86_64 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:44.174 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-09-22T03:41:44.533 INFO:tasks.cbt:benchmark: librbdfio 2024-09-22T03:41:44.533 DEBUG:teuthology.orchestra.run.smithi174:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-09-22T03:41:44.548 INFO:teuthology.orchestra.run.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-09-22T03:41:46.779 DEBUG:teuthology.orchestra.run.smithi174:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-09-22T03:41:47.337 INFO:teuthology.orchestra.run.smithi174.stdout:Operating system Linux 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout:CPU x86_64 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout:Big endian no 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout:Compiler gcc 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout:Cross compile no 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-09-22T03:41:47.338 INFO:teuthology.orchestra.run.smithi174.stdout:Static build no 2024-09-22T03:41:47.466 INFO:teuthology.orchestra.run.smithi174.stdout:Wordsize 64 2024-09-22T03:41:47.474 INFO:teuthology.orchestra.run.smithi174.stdout:zlib no 2024-09-22T03:41:47.515 INFO:teuthology.orchestra.run.smithi174.stdout:fcntl(F_FULLFSYNC) no 2024-09-22T03:41:47.530 INFO:teuthology.orchestra.run.smithi174.stdout:Linux AIO support no 2024-09-22T03:41:47.530 INFO:teuthology.orchestra.run.smithi174.stdout:Linux AIO support rw flags no 2024-09-22T03:41:47.598 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX AIO support yes 2024-09-22T03:41:47.598 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX AIO support needs -lrt no 2024-09-22T03:41:47.629 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX AIO fsync yes 2024-09-22T03:41:47.659 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX pshared support yes 2024-09-22T03:41:47.711 INFO:teuthology.orchestra.run.smithi174.stdout:pthread_condattr_setclock() yes 2024-09-22T03:41:47.758 INFO:teuthology.orchestra.run.smithi174.stdout:pthread_sigmask() yes 2024-09-22T03:41:47.794 INFO:teuthology.orchestra.run.smithi174.stdout:pthread_getaffinity_np() yes 2024-09-22T03:41:47.803 INFO:teuthology.orchestra.run.smithi174.stdout:Solaris AIO support no 2024-09-22T03:41:47.850 INFO:teuthology.orchestra.run.smithi174.stdout:__sync_fetch_and_add yes 2024-09-22T03:41:47.896 INFO:teuthology.orchestra.run.smithi174.stdout:__sync_synchronize yes 2024-09-22T03:41:47.925 INFO:teuthology.orchestra.run.smithi174.stdout:__sync_val_compare_and_swap yes 2024-09-22T03:41:47.934 INFO:teuthology.orchestra.run.smithi174.stdout:libverbs no 2024-09-22T03:41:47.958 INFO:teuthology.orchestra.run.smithi174.stdout:rdmacm no 2024-09-22T03:41:47.989 INFO:teuthology.orchestra.run.smithi174.stdout:asprintf() yes 2024-09-22T03:41:48.024 INFO:teuthology.orchestra.run.smithi174.stdout:vasprintf() yes 2024-09-22T03:41:48.056 INFO:teuthology.orchestra.run.smithi174.stdout:Linux fallocate yes 2024-09-22T03:41:48.088 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX fadvise yes 2024-09-22T03:41:48.120 INFO:teuthology.orchestra.run.smithi174.stdout:POSIX fallocate yes 2024-09-22T03:41:48.163 INFO:teuthology.orchestra.run.smithi174.stdout:sched_setaffinity(3 arg) yes 2024-09-22T03:41:48.163 INFO:teuthology.orchestra.run.smithi174.stdout:sched_setaffinity(2 arg) no 2024-09-22T03:41:48.195 INFO:teuthology.orchestra.run.smithi174.stdout:clock_gettime yes 2024-09-22T03:41:48.226 INFO:teuthology.orchestra.run.smithi174.stdout:CLOCK_MONOTONIC yes 2024-09-22T03:41:48.258 INFO:teuthology.orchestra.run.smithi174.stdout:clockid_t yes 2024-09-22T03:41:48.300 INFO:teuthology.orchestra.run.smithi174.stdout:gettimeofday yes 2024-09-22T03:41:48.332 INFO:teuthology.orchestra.run.smithi174.stdout:fdatasync yes 2024-09-22T03:41:48.363 INFO:teuthology.orchestra.run.smithi174.stdout:pipe() yes 2024-09-22T03:41:48.394 INFO:teuthology.orchestra.run.smithi174.stdout:pipe2() yes 2024-09-22T03:41:48.426 INFO:teuthology.orchestra.run.smithi174.stdout:pread() yes 2024-09-22T03:41:48.505 INFO:teuthology.orchestra.run.smithi174.stdout:sync_file_range yes 2024-09-22T03:41:48.513 INFO:teuthology.orchestra.run.smithi174.stdout:ASharedMemory_create no 2024-09-22T03:41:48.525 INFO:teuthology.orchestra.run.smithi174.stdout:EXT4 move extent yes 2024-09-22T03:41:48.556 INFO:teuthology.orchestra.run.smithi174.stdout:Linux splice(2) yes 2024-09-22T03:41:48.565 INFO:teuthology.orchestra.run.smithi174.stdout:libnuma no 2024-09-22T03:41:48.594 INFO:teuthology.orchestra.run.smithi174.stdout:strsep yes 2024-09-22T03:41:48.624 INFO:teuthology.orchestra.run.smithi174.stdout:strcasestr yes 2024-09-22T03:41:48.638 INFO:teuthology.orchestra.run.smithi174.stdout:strlcat no 2024-09-22T03:41:48.673 INFO:teuthology.orchestra.run.smithi174.stdout:getopt_long_only() yes 2024-09-22T03:41:48.741 INFO:teuthology.orchestra.run.smithi174.stdout:inet_aton yes 2024-09-22T03:41:48.772 INFO:teuthology.orchestra.run.smithi174.stdout:socklen_t yes 2024-09-22T03:41:48.805 INFO:teuthology.orchestra.run.smithi174.stdout:__thread yes 2024-09-22T03:41:48.842 INFO:teuthology.orchestra.run.smithi174.stdout:RUSAGE_THREAD yes 2024-09-22T03:41:48.872 INFO:teuthology.orchestra.run.smithi174.stdout:SCHED_IDLE yes 2024-09-22T03:41:48.908 INFO:teuthology.orchestra.run.smithi174.stdout:TCP_NODELAY yes 2024-09-22T03:41:48.961 INFO:teuthology.orchestra.run.smithi174.stdout:vsock yes 2024-09-22T03:41:48.995 INFO:teuthology.orchestra.run.smithi174.stdout:Net engine window_size yes 2024-09-22T03:41:49.029 INFO:teuthology.orchestra.run.smithi174.stdout:TCP_MAXSEG yes 2024-09-22T03:41:49.059 INFO:teuthology.orchestra.run.smithi174.stdout:RLIMIT_MEMLOCK yes 2024-09-22T03:41:49.093 INFO:teuthology.orchestra.run.smithi174.stdout:pwritev/preadv yes 2024-09-22T03:41:49.127 INFO:teuthology.orchestra.run.smithi174.stdout:pwritev2/preadv2 yes 2024-09-22T03:41:49.164 INFO:teuthology.orchestra.run.smithi174.stdout:IPv6 helpers yes 2024-09-22T03:41:49.181 INFO:teuthology.orchestra.run.smithi174.stdout:http engine no 2024-09-22T03:41:49.325 INFO:teuthology.orchestra.run.smithi174.stdout:Rados engine yes 2024-09-22T03:41:49.508 INFO:teuthology.orchestra.run.smithi174.stdout:Rados Block Device engine yes 2024-09-22T03:41:49.693 INFO:teuthology.orchestra.run.smithi174.stdout:rbd_poll yes 2024-09-22T03:41:49.876 INFO:teuthology.orchestra.run.smithi174.stdout:rbd_invalidate_cache yes 2024-09-22T03:41:49.907 INFO:teuthology.orchestra.run.smithi174.stdout:setvbuf yes 2024-09-22T03:41:49.916 INFO:teuthology.orchestra.run.smithi174.stdout:Gluster API engine no 2024-09-22T03:41:49.932 INFO:teuthology.orchestra.run.smithi174.stdout:s390_z196_facilities no 2024-09-22T03:41:49.933 INFO:teuthology.orchestra.run.smithi174.stdout:HDFS engine no 2024-09-22T03:41:49.979 INFO:teuthology.orchestra.run.smithi174.stdout:MTD yes 2024-09-22T03:41:49.987 INFO:teuthology.orchestra.run.smithi174.stdout:libpmem no 2024-09-22T03:41:49.987 INFO:teuthology.orchestra.run.smithi174.stdout:libpmem1_5 no 2024-09-22T03:41:49.995 INFO:teuthology.orchestra.run.smithi174.stdout:libpmem2 no 2024-09-22T03:41:49.995 INFO:teuthology.orchestra.run.smithi174.stdout:PMDK dev-dax engine no 2024-09-22T03:41:49.995 INFO:teuthology.orchestra.run.smithi174.stdout:PMDK libpmem engine no 2024-09-22T03:41:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:DDN's Infinite Memory Engine no 2024-09-22T03:41:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:iscsi engine no 2024-09-22T03:41:50.003 INFO:teuthology.orchestra.run.smithi174.stdout:NBD engine no 2024-09-22T03:41:50.012 INFO:teuthology.orchestra.run.smithi174.stdout:DAOS File System (dfs) Engine no 2024-09-22T03:41:50.021 INFO:teuthology.orchestra.run.smithi174.stdout:NFS engine no 2024-09-22T03:41:50.021 INFO:teuthology.orchestra.run.smithi174.stdout:lex/yacc for arithmetic no 2024-09-22T03:41:50.053 INFO:teuthology.orchestra.run.smithi174.stdout:getmntent yes 2024-09-22T03:41:50.079 INFO:teuthology.orchestra.run.smithi174.stdout:getmntinfo no 2024-09-22T03:41:50.138 INFO:teuthology.orchestra.run.smithi174.stdout:Static Assert yes 2024-09-22T03:41:50.166 INFO:teuthology.orchestra.run.smithi174.stdout:bool yes 2024-09-22T03:41:50.200 INFO:teuthology.orchestra.run.smithi174.stdout:strndup yes 2024-09-22T03:41:50.207 INFO:teuthology.orchestra.run.smithi174.stdout:Valgrind headers no 2024-09-22T03:41:50.237 INFO:teuthology.orchestra.run.smithi174.stdout:Zoned block device support yes 2024-09-22T03:41:50.265 INFO:teuthology.orchestra.run.smithi174.stdout:Zoned block device capacity yes 2024-09-22T03:41:50.273 INFO:teuthology.orchestra.run.smithi174.stdout:libzbc engine no 2024-09-22T03:41:50.309 INFO:teuthology.orchestra.run.smithi174.stdout:NVMe uring command support yes 2024-09-22T03:41:50.324 INFO:teuthology.orchestra.run.smithi174.stdout:xnvme engine no 2024-09-22T03:41:50.331 INFO:teuthology.orchestra.run.smithi174.stdout:isal no 2024-09-22T03:41:50.346 INFO:teuthology.orchestra.run.smithi174.stdout:libblkio engine no 2024-09-22T03:41:50.346 INFO:teuthology.orchestra.run.smithi174.stdout:march_armv8_a_crc_crypto no 2024-09-22T03:41:50.346 INFO:teuthology.orchestra.run.smithi174.stdout:cuda no 2024-09-22T03:41:50.347 INFO:teuthology.orchestra.run.smithi174.stdout:libcufile no 2024-09-22T03:41:50.374 INFO:teuthology.orchestra.run.smithi174.stdout:Build march=native yes 2024-09-22T03:41:50.382 INFO:teuthology.orchestra.run.smithi174.stdout:CUnit no 2024-09-22T03:41:50.411 INFO:teuthology.orchestra.run.smithi174.stdout:__kernel_rwf_t yes 2024-09-22T03:41:50.438 INFO:teuthology.orchestra.run.smithi174.stdout:-Wimplicit-fallthrough=2 yes 2024-09-22T03:41:50.470 INFO:teuthology.orchestra.run.smithi174.stdout:-Wno-stringop-truncation yes 2024-09-22T03:41:50.507 INFO:teuthology.orchestra.run.smithi174.stdout:MADV_HUGEPAGE yes 2024-09-22T03:41:50.538 INFO:teuthology.orchestra.run.smithi174.stdout:gettid yes 2024-09-22T03:41:50.573 INFO:teuthology.orchestra.run.smithi174.stdout:statx(2)/libc yes 2024-09-22T03:41:50.607 INFO:teuthology.orchestra.run.smithi174.stdout:statx(2)/syscall yes 2024-09-22T03:41:50.610 INFO:teuthology.orchestra.run.smithi174.stdout:Windows PDB generation no 2024-09-22T03:41:50.641 INFO:teuthology.orchestra.run.smithi174.stdout:timerfd_create yes 2024-09-22T03:41:50.643 INFO:teuthology.orchestra.run.smithi174.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-09-22T03:41:50.643 INFO:teuthology.orchestra.run.smithi174.stdout:Lib-based ioengines dynamic no 2024-09-22T03:41:50.706 INFO:teuthology.orchestra.run.smithi174.stdout:TCMalloc support yes 2024-09-22T03:41:50.707 INFO:teuthology.orchestra.run.smithi174.stdout:seed_buckets 4 2024-09-22T03:41:50.790 INFO:teuthology.orchestra.run.smithi174.stderr:FIO_VERSION = fio-3.37-124-gbcd4 2024-09-22T03:41:50.824 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc16.o 2024-09-22T03:41:50.904 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc32.o 2024-09-22T03:41:50.942 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc32c-arm64.o 2024-09-22T03:41:51.024 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc32c-intel.o 2024-09-22T03:41:51.070 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc32c.o 2024-09-22T03:41:51.112 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc64.o 2024-09-22T03:41:51.150 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crc7.o 2024-09-22T03:41:51.185 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/crct10dif_common.o 2024-09-22T03:41:51.223 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/fnv.o 2024-09-22T03:41:51.267 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/md5.o 2024-09-22T03:41:51.380 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/murmur3.o 2024-09-22T03:41:51.423 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/sha1.o 2024-09-22T03:41:51.611 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/sha256.o 2024-09-22T03:41:51.905 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/sha3.o 2024-09-22T03:41:52.050 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/sha512.o 2024-09-22T03:41:52.161 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/test.o 2024-09-22T03:41:52.332 INFO:teuthology.orchestra.run.smithi174.stdout: CC crc/xxhash.o 2024-09-22T03:41:52.428 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/axmap.o 2024-09-22T03:41:52.552 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/bloom.o 2024-09-22T03:41:52.637 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/flist_sort.o 2024-09-22T03:41:52.709 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/gauss.o 2024-09-22T03:41:52.811 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/getrusage.o 2024-09-22T03:41:52.847 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/hweight.o 2024-09-22T03:41:52.886 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/ieee754.o 2024-09-22T03:41:52.936 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/lfsr.o 2024-09-22T03:41:53.038 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/memalign.o 2024-09-22T03:41:53.084 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/memcpy.o 2024-09-22T03:41:53.230 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/mountcheck.o 2024-09-22T03:41:53.272 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/num2str.o 2024-09-22T03:41:53.339 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/output_buffer.o 2024-09-22T03:41:53.387 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/pattern.o 2024-09-22T03:41:53.529 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/prio_tree.o 2024-09-22T03:41:53.649 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/rand.o 2024-09-22T03:41:53.836 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/rbtree.o 2024-09-22T03:41:53.960 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/strntol.o 2024-09-22T03:41:54.009 INFO:teuthology.orchestra.run.smithi174.stdout: CC lib/zipf.o 2024-09-22T03:41:54.129 INFO:teuthology.orchestra.run.smithi174.stdout: CC gettime.o 2024-09-22T03:41:54.423 INFO:teuthology.orchestra.run.smithi174.stdout: CC ioengines.o 2024-09-22T03:41:54.691 INFO:teuthology.orchestra.run.smithi174.stdout: CC init.o 2024-09-22T03:41:55.463 INFO:teuthology.orchestra.run.smithi174.stdout: CC stat.o 2024-09-22T03:41:57.289 INFO:teuthology.orchestra.run.smithi174.stdout: CC log.o 2024-09-22T03:41:57.414 INFO:teuthology.orchestra.run.smithi174.stdout: CC time.o 2024-09-22T03:41:57.540 INFO:teuthology.orchestra.run.smithi174.stdout: CC filesetup.o 2024-09-22T03:41:58.232 INFO:teuthology.orchestra.run.smithi174.stdout: CC eta.o 2024-09-22T03:41:58.547 INFO:teuthology.orchestra.run.smithi174.stdout: CC verify.o 2024-09-22T03:41:59.233 INFO:teuthology.orchestra.run.smithi174.stdout: CC memory.o 2024-09-22T03:41:59.417 INFO:teuthology.orchestra.run.smithi174.stdout: CC io_u.o 2024-09-22T03:42:00.191 INFO:teuthology.orchestra.run.smithi174.stdout: CC parse.o 2024-09-22T03:42:00.659 INFO:teuthology.orchestra.run.smithi174.stdout: CC fio_sem.o 2024-09-22T03:42:00.764 INFO:teuthology.orchestra.run.smithi174.stdout: CC rwlock.o 2024-09-22T03:42:00.849 INFO:teuthology.orchestra.run.smithi174.stdout: CC pshared.o 2024-09-22T03:42:00.914 INFO:teuthology.orchestra.run.smithi174.stdout: CC options.o 2024-09-22T03:42:01.720 INFO:teuthology.orchestra.run.smithi174.stdout: CC smalloc.o 2024-09-22T03:42:01.907 INFO:teuthology.orchestra.run.smithi174.stdout: CC filehash.o 2024-09-22T03:42:02.064 INFO:teuthology.orchestra.run.smithi174.stdout: CC profile.o 2024-09-22T03:42:02.183 INFO:teuthology.orchestra.run.smithi174.stdout: CC debug.o 2024-09-22T03:42:02.226 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/cpu.o 2024-09-22T03:42:02.364 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/mmap.o 2024-09-22T03:42:02.532 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/sync.o 2024-09-22T03:42:02.760 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/null.o 2024-09-22T03:42:02.878 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/net.o 2024-09-22T03:42:03.319 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/ftruncate.o 2024-09-22T03:42:03.417 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/fileoperations.o 2024-09-22T03:42:03.571 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/exec.o 2024-09-22T03:42:03.723 INFO:teuthology.orchestra.run.smithi174.stdout: CC server.o 2024-09-22T03:42:04.463 INFO:teuthology.orchestra.run.smithi174.stdout: CC client.o 2024-09-22T03:42:04.999 INFO:teuthology.orchestra.run.smithi174.stdout: CC iolog.o 2024-09-22T03:42:05.459 INFO:teuthology.orchestra.run.smithi174.stdout: CC backend.o 2024-09-22T03:42:06.308 INFO:teuthology.orchestra.run.smithi174.stdout: CC libfio.o 2024-09-22T03:42:06.478 INFO:teuthology.orchestra.run.smithi174.stdout: CC flow.o 2024-09-22T03:42:06.592 INFO:teuthology.orchestra.run.smithi174.stdout: CC cconv.o 2024-09-22T03:42:06.945 INFO:teuthology.orchestra.run.smithi174.stdout: CC gettime-thread.o 2024-09-22T03:42:07.051 INFO:teuthology.orchestra.run.smithi174.stdout: CC helpers.o 2024-09-22T03:42:07.113 INFO:teuthology.orchestra.run.smithi174.stdout: CC json.o 2024-09-22T03:42:07.308 INFO:teuthology.orchestra.run.smithi174.stdout: CC idletime.o 2024-09-22T03:42:07.531 INFO:teuthology.orchestra.run.smithi174.stdout: CC td_error.o 2024-09-22T03:42:07.628 INFO:teuthology.orchestra.run.smithi174.stdout: CC profiles/tiobench.o 2024-09-22T03:42:07.724 INFO:teuthology.orchestra.run.smithi174.stdout: CC profiles/act.o 2024-09-22T03:42:07.883 INFO:teuthology.orchestra.run.smithi174.stdout: CC io_u_queue.o 2024-09-22T03:42:07.935 INFO:teuthology.orchestra.run.smithi174.stdout: CC filelock.o 2024-09-22T03:42:08.061 INFO:teuthology.orchestra.run.smithi174.stdout: CC workqueue.o 2024-09-22T03:42:08.241 INFO:teuthology.orchestra.run.smithi174.stdout: CC rate-submit.o 2024-09-22T03:42:08.390 INFO:teuthology.orchestra.run.smithi174.stdout: CC optgroup.o 2024-09-22T03:42:08.436 INFO:teuthology.orchestra.run.smithi174.stdout: CC helper_thread.o 2024-09-22T03:42:08.615 INFO:teuthology.orchestra.run.smithi174.stdout: CC steadystate.o 2024-09-22T03:42:08.854 INFO:teuthology.orchestra.run.smithi174.stdout: CC zone-dist.o 2024-09-22T03:42:08.989 INFO:teuthology.orchestra.run.smithi174.stdout: CC zbd.o 2024-09-22T03:42:09.688 INFO:teuthology.orchestra.run.smithi174.stdout: CC dedupe.o 2024-09-22T03:42:09.833 INFO:teuthology.orchestra.run.smithi174.stdout: CC dataplacement.o 2024-09-22T03:42:10.003 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/posixaio.o 2024-09-22T03:42:10.139 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/falloc.o 2024-09-22T03:42:10.254 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/e4defrag.o 2024-09-22T03:42:10.386 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/splice.o 2024-09-22T03:42:10.574 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/asprintf.o 2024-09-22T03:42:10.611 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/strlcat.o 2024-09-22T03:42:10.655 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/mtd.o 2024-09-22T03:42:10.820 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/libmtd.o 2024-09-22T03:42:11.200 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/libmtd_legacy.o 2024-09-22T03:42:11.334 INFO:teuthology.orchestra.run.smithi174.stdout: CC diskutil.o 2024-09-22T03:42:11.547 INFO:teuthology.orchestra.run.smithi174.stdout: CC fifo.o 2024-09-22T03:42:11.602 INFO:teuthology.orchestra.run.smithi174.stdout: CC blktrace.o 2024-09-22T03:42:11.932 INFO:teuthology.orchestra.run.smithi174.stdout: CC cgroup.o 2024-09-22T03:42:12.084 INFO:teuthology.orchestra.run.smithi174.stdout: CC trim.o 2024-09-22T03:42:12.201 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/sg.o 2024-09-22T03:42:12.588 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/linux-dev-lookup.o 2024-09-22T03:42:12.647 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/io_uring.o 2024-09-22T03:42:13.041 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/nvme.o 2024-09-22T03:42:13.355 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/linux-blkzoned.o 2024-09-22T03:42:13.508 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/cmdprio.o 2024-09-22T03:42:13.746 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/rados.o 2024-09-22T03:42:13.922 INFO:teuthology.orchestra.run.smithi174.stdout: CC engines/rbd.o 2024-09-22T03:42:14.103 INFO:teuthology.orchestra.run.smithi174.stdout: CC fio.o 2024-09-22T03:42:14.196 INFO:teuthology.orchestra.run.smithi174.stdout: LINK fio 2024-09-22T03:42:14.472 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/log.o 2024-09-22T03:42:14.516 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/genzipf.o 2024-09-22T03:42:14.653 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/strcasestr.o 2024-09-22T03:42:14.679 INFO:teuthology.orchestra.run.smithi174.stdout: CC oslib/strndup.o 2024-09-22T03:42:14.703 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/fio-genzipf 2024-09-22T03:42:14.876 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/btrace2fio.o 2024-09-22T03:42:15.307 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/fio-btrace2fio 2024-09-22T03:42:15.485 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/verify-state.o 2024-09-22T03:42:15.584 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/debug.o 2024-09-22T03:42:15.620 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/fio-verify-state 2024-09-22T03:42:15.791 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/stest.o 2024-09-22T03:42:15.868 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/arch.o 2024-09-22T03:42:15.899 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/stest 2024-09-22T03:42:16.072 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/ieee754.o 2024-09-22T03:42:16.112 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/ieee754 2024-09-22T03:42:16.285 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/axmap.o 2024-09-22T03:42:16.388 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/axmap 2024-09-22T03:42:16.559 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/lfsr-test.o 2024-09-22T03:42:16.657 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/lfsr-test 2024-09-22T03:42:16.828 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/gen-rand.o 2024-09-22T03:42:16.928 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/gen-rand 2024-09-22T03:42:17.103 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/memlock.o 2024-09-22T03:42:17.165 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/memlock 2024-09-22T03:42:17.339 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/read-to-pipe-async.o 2024-09-22T03:42:17.524 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/read-to-pipe-async 2024-09-22T03:42:17.696 INFO:teuthology.orchestra.run.smithi174.stdout: CC t/io_uring.o 2024-09-22T03:42:18.235 INFO:teuthology.orchestra.run.smithi174.stdout: LINK t/io_uring 2024-09-22T03:42:18.408 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-22T03:42:18.434 DEBUG:teuthology.orchestra.run.smithi174:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - Settings.cluster: 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'clients': ['ubuntu@smithi174.front.sepia.ceph.com'], 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'head': 'ubuntu@smithi174.front.sepia.ceph.com', 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'is_teuthology': True, 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'iterations': 1, 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'mons': ['ubuntu@smithi174.front.sepia.ceph.com'], 2024-09-22T03:42:18.683 INFO:teuthology.orchestra.run.smithi174.stderr: 'osds': ['ubuntu@smithi174.front.sepia.ceph.com'], 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'osds_per_node': 3, 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'pid_dir': '/var/run/ceph', 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'rebuild_every_test': False, 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'rgws': {'ubuntu@smithi174.front.sepia.ceph.com': None}, 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'use_existing': True, 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr: 'user': 'ubuntu'} 2024-09-22T03:42:18.684 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd 2024-09-22T03:42:18.685 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite; fi 2024-09-22T03:42:18.686 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite 2024-09-22T03:42:18.689 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-22T03:42:18.689 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl 2024-09-22T03:42:18.692 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:42:18.693 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/perf 2024-09-22T03:42:18.696 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:42:18.696 INFO:teuthology.orchestra.run.smithi174.stderr:03:42:18 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:43:18.758 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:43:18.815 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:43:18.865 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* 2024-09-22T03:43:18.879 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7da5bbcd/$(basename ${i}).smithi174.front.sepia.ceph.com ; done 2024-09-22T03:43:18.883 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - INFO - cbt - Creating fio images... 2024-09-22T03:43:18.883 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl 2024-09-22T03:43:18.886 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:43:18.887 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/perf 2024-09-22T03:43:18.890 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:43:18.890 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:43:18.890 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-09-22T03:43:19.210 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-09-22T03:43:20.179 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-09-22T03:43:21.181 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:21 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-09-22T03:43:22.197 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:22 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-09-22T03:43:23.194 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:23 - INFO - cbt - Checking Health after pool creation. 2024-09-22T03:43:28.199 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:43:28.663 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:28 - INFO - cbt - Final Pool Health Check. 2024-09-22T03:43:33.668 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:43:34.038 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-09-22T03:43:34.041 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:34 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi174.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-09-22T03:43:35.590 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi174.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-09-22T03:43:35.762 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:43:35.795 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:43:35.838 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - INFO - cbt - Attempting to prefill fio images... 2024-09-22T03:43:35.838 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-22T03:43:35.840 INFO:teuthology.orchestra.run.smithi174.stderr:03:43:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-09-22T03:49:00.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:49:06.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2024-09-22T03:49:11.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~11s 2024-09-22T03:49:17.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~16s 2024-09-22T03:49:22.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~22s 2024-09-22T03:49:27.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~27s 2024-09-22T03:49:33.238 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~32s 2024-09-22T03:49:38.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~38s 2024-09-22T03:49:44.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~43s 2024-09-22T03:49:49.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~49s 2024-09-22T03:49:54.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~54s 2024-09-22T03:49:55.481 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:49:55.479+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:00.253 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~59s 2024-09-22T03:50:00.482 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:00.481+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:05.483 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:05.482+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:05.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~65s 2024-09-22T03:50:10.484 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:10.483+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:11.060 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~70s 2024-09-22T03:50:15.484 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:15.483+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:16.462 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~76s 2024-09-22T03:50:20.485 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:20.484+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:21.865 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~81s 2024-09-22T03:50:25.485 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:25.484+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:27.268 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~86s 2024-09-22T03:50:30.486 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:30.484+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:32.673 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~92s 2024-09-22T03:50:35.486 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:35.485+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:38.076 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~97s 2024-09-22T03:50:40.486 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:40.484+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:43.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~103s 2024-09-22T03:50:45.487 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:45.485+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:48.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~108s 2024-09-22T03:50:50.487 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:50.485+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:54.287 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~113s 2024-09-22T03:50:55.488 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:50:55.486+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:50:59.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~119s 2024-09-22T03:51:00.488 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:00.487+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:05.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~124s 2024-09-22T03:51:05.488 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:05.487+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:10.489 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:10.488+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:10.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~130s 2024-09-22T03:51:15.489 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:15.488+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:15.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~135s 2024-09-22T03:51:20.490 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:20.489+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:21.301 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~140s 2024-09-22T03:51:25.490 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:25.489+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:26.705 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~146s 2024-09-22T03:51:30.491 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:30.490+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:32.108 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~151s 2024-09-22T03:51:35.491 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:35.490+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:37.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~157s 2024-09-22T03:51:40.491 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:40.490+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:42.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~162s 2024-09-22T03:51:45.492 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:45.491+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:48.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~167s 2024-09-22T03:51:50.492 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:50.490+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:53.720 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~173s 2024-09-22T03:51:55.493 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:51:55.491+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:51:59.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~178s 2024-09-22T03:52:00.493 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:00.491+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:04.527 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~184s 2024-09-22T03:52:05.494 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:05.492+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:09.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~189s 2024-09-22T03:52:10.494 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:10.492+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:15.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~195s 2024-09-22T03:52:15.495 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:15.493+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:20.495 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:20.494+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:20.736 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~200s 2024-09-22T03:52:25.496 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:25.495+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:26.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~205s 2024-09-22T03:52:30.496 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:30.495+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:31.542 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~211s 2024-09-22T03:52:35.497 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:35.496+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:36.944 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~216s 2024-09-22T03:52:40.497 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:40.496+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:42.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~222s 2024-09-22T03:52:45.497 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:45.496+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:47.750 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~227s 2024-09-22T03:52:50.498 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:50.497+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:53.153 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~232s 2024-09-22T03:52:55.498 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:52:55.497+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:52:58.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~238s 2024-09-22T03:53:00.499 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:00.498+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:03.959 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~243s 2024-09-22T03:53:05.499 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:05.497+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:09.362 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~249s 2024-09-22T03:53:10.500 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:10.498+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:14.765 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~254s 2024-09-22T03:53:15.500 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:15.498+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:20.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~259s 2024-09-22T03:53:20.501 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:20.499+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:25.501 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:25.499+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:25.571 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~265s 2024-09-22T03:53:30.502 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:30.500+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:30.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~270s 2024-09-22T03:53:35.502 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:35.501+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:36.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~276s 2024-09-22T03:53:40.502 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:40.501+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:41.780 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~281s 2024-09-22T03:53:45.503 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:45.502+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:47.183 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~286s 2024-09-22T03:53:50.503 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:50.502+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:52.586 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~292s 2024-09-22T03:53:55.504 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:53:55.503+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:53:57.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~297s 2024-09-22T03:54:00.504 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:54:00.503+0000 7feb00ff9640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 v2:172.21.15.174:6805/3033857583 for 30sec e28 v28) 2024-09-22T03:54:03.393 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~303s 2024-09-22T03:54:03.393 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-09-22T03:54:03.795 INFO:tasks.ceph.osd.0:Sent signal 15 2024-09-22T03:54:03.796 INFO:tasks.ceph.osd.2:Sent signal 15 2024-09-22T03:54:03.796 INFO:tasks.ceph.mon.a:Sent signal 15 2024-09-22T03:54:03.796 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-09-22T03:54:03.797 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:54:03.796+0000 7feb03fff640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 28486) UID: 0 2024-09-22T03:54:03.798 INFO:tasks.ceph.mon.a.smithi174.stderr:2024-09-22T03:54:03.796+0000 7feb03fff640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-22T03:54:03.798 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:54:03.796+0000 7fa0fe5f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 28489) UID: 0 2024-09-22T03:54:03.798 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-09-22T03:54:03.796+0000 7fa0fe5f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-22T11:31:56.731 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-09-22T11:31:56.761 DEBUG:teuthology.task.console_log:Killing console logger for smithi174 2024-09-22T11:31:56.762 DEBUG:teuthology.exit:Finished running handlers