2024-09-22T03:55:22.705 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:55:22.725 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:55:22.815 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406 branch: main description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914406' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore ompap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2950 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi052.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBaikvqvxRnYyPC5OuJ2Mk9oe2O/HKR+HJsUR4+aXu2IFjGYrFUPM3qTgJ84PjjOBC1EdxKarea92BkIAcvbp6g= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_20:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:55:22.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-22T03:55:22.816 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-22T03:55:22.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:55:22.818 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:55:22.839 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-22T03:55:22.839 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:55:22.839 INFO:teuthology.packaging:ref: None 2024-09-22T03:55:22.839 INFO:teuthology.packaging:tag: None 2024-09-22T03:55:22.839 INFO:teuthology.packaging:branch: main 2024-09-22T03:55:22.839 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T03:55:22.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=main 2024-09-22T03:55:23.039 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-22T03:55:23.040 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:55:23.047 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:55:23.047 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:55:23.053 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:55:23.069 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:55:23.076 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:55:23.101 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi052.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:50:46.740075', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBaikvqvxRnYyPC5OuJ2Mk9oe2O/HKR+HJsUR4+aXu2IFjGYrFUPM3qTgJ84PjjOBC1EdxKarea92BkIAcvbp6g='} 2024-09-22T03:55:23.101 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:55:23.107 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:55:23.107 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:55:23.174 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2634dd0160>, signals=[15]) 2024-09-22T03:55:23.174 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:55:23.180 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:55:23.181 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T03:55:23.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:55:23.259 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:55:23.266 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2024-09-22T03:55:23.344 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2024-09-22T03:55:23.345 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="CentOS Stream" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="9" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:ID="centos" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="9" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2024-09-22T03:55:23.403 INFO:teuthology.orchestra.run.smithi052.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:55:23.404 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:55:23.404 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:55:23.404 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:55:23.404 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:55:23.404 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:55:23.404 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2024-09-22T03:55:23.430 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:55:23.438 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:55:23.443 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:55:23.443 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:55:23.461 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:55:23.466 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:55:23.466 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:55:23.530 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:55:23.567 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:55:23.581 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:55:23.581 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:55:23.582 DEBUG:teuthology.orchestra.run.smithi052:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:55:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:55:23.600 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-22T03:55:23.658 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:55:23.658 INFO:teuthology.task.kernel:Running kernel on smithi052: 5.14.0-503.el9.x86_64 2024-09-22T03:55:23.659 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum install -y kernel 2024-09-22T03:55:27.307 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-22T03:55:30.491 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-09-22T03:55:36.075 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.5 MB 00:01 2024-09-22T03:55:37.542 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 110 kB/s | 19 kB 00:00 2024-09-22T03:55:38.613 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-22T03:55:44.135 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-09-22T03:55:46.184 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:55:46.184 INFO:teuthology.orchestra.run.smithi052.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:55:46.286 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T03:55:46.296 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T03:55:46.297 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T03:55:46.298 INFO:teuthology.orchestra.run.smithi052.stdout:Install 4 Packages 2024-09-22T03:55:46.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:55:46.298 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 79 M 2024-09-22T03:55:46.298 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 126 M 2024-09-22T03:55:46.298 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T03:55:46.495 INFO:teuthology.orchestra.run.smithi052.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 460 kB/s | 36 kB 00:00 2024-09-22T03:55:47.862 INFO:teuthology.orchestra.run.smithi052.stdout:(2/4): kernel-modules-core-5.14.0-508.el9.x86_6 21 MB/s | 29 MB 00:01 2024-09-22T03:55:48.454 INFO:teuthology.orchestra.run.smithi052.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-22T03:55:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 6.6 MB/s | 16 MB 00:02 2024-09-22T03:55:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:55:48.779 INFO:teuthology.orchestra.run.smithi052.stdout:Total 32 MB/s | 79 MB 00:02 2024-09-22T03:55:49.485 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T03:55:49.876 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T03:55:49.876 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T03:55:50.867 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T03:55:50.867 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T03:55:53.436 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T03:55:54.099 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:55:54.197 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:55:56.958 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:55:57.084 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:56:02.213 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:56:02.252 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:07.403 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:46.438 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:46.479 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:56:46.840 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:56:46.841 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:56:47.644 INFO:teuthology.orchestra.run.smithi052.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:56:47.645 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:56:47.645 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T03:56:47.981 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:56:48.057 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:56:48.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:56:48.057 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:56:48.057 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:56:48.058 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:56:48.058 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:56:48.058 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:56:48.058 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:56:48.116 INFO:teuthology.orchestra.run.smithi052.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:56:48.117 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:56:48.117 DEBUG:teuthology.orchestra.run.smithi052:> sudo rpm -qi grub2-tools 2024-09-22T03:56:48.148 INFO:teuthology.orchestra.run.smithi052.stdout:Name : grub2-tools 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Epoch : 1 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Version : 2.06 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Release : 82.el9 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: x86_64 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Group : Unspecified 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Size : 8274670 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:License : GPLv3+ 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Packager : builder@centos.org 2024-09-22T03:56:48.149 INFO:teuthology.orchestra.run.smithi052.stdout:Vendor : CentOS 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:Summary : Support tools for GRUB. 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:Description : 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:hardware devices. 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T03:56:48.150 INFO:teuthology.orchestra.run.smithi052.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:56:48.151 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:56:48.151 INFO:teuthology.task.kernel:Updating grub on smithi052 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:56:48.151 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:56:48.834 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2024-09-22T03:56:50.083 INFO:teuthology.orchestra.run.smithi052.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:56:50.110 INFO:teuthology.orchestra.run.smithi052.stderr:done 2024-09-22T03:56:50.112 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-09-22T03:56:50.127 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.gle57Us6wS 2024-09-22T03:56:50.127 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gle57Us6wS 2024-09-22T03:56:50.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.gle57Us6wS 2024-09-22T03:56:50.330 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.gle57Us6wS is 6KB 2024-09-22T03:56:50.378 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.gle57Us6wS 2024-09-22T03:56:50.393 DEBUG:teuthology.orchestra.run.smithi052:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:56:50.456 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:56:50.456 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:56:50.456 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:56:50.456 INFO:teuthology.orchestra.run.smithi052.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:56:50.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:56:50.612 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2024-09-22T03:57:20.642 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:57:20.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T03:57:20.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:57:39.183 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.52 2024-09-22T03:57:48.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T03:57:48.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:58:48.250 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:59:00.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T03:59:00.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:59:00.537 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-22T03:59:00.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T03:59:00.930 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:59:00.930 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2024-09-22T03:59:00.949 INFO:teuthology.orchestra.run.smithi052.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:59:00.949 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:59:00.949 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:59:00.949 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:59:01.950 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:59:01.950 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:59:02.014 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2024-09-22T03:59:02.041 DEBUG:teuthology.parallel:result is None 2024-09-22T03:59:02.041 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:59:02.050 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:59:02.051 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:59:02.069 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:59:02.077 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:59:02.089 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:59:02.090 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:59:02.137 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:59:02.145 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:59:02.146 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:59:02.211 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:59:02.224 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:59:02.225 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:59:02.233 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:59:02.233 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:59:02.295 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:59:02.304 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:59:02.304 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:59:02.354 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:59:02.437 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:59:02.491 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:59:02.575 DEBUG:teuthology.orchestra.run.smithi052:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:59:02.622 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T03:59:02.622 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:59:02.690 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2024-09-22T03:59:02.772 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:59:03.160 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:59:03.174 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:59:03.174 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:59:03.185 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:59:03.195 DEBUG:teuthology.orchestra.run.smithi052:> sudo service auditd rotate 2024-09-22T03:59:03.264 INFO:teuthology.orchestra.run.smithi052.stdout:Rotating logs: 2024-09-22T03:59:03.266 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:59:03.266 DEBUG:teuthology.orchestra.run.smithi052:> /usr/sbin/getenforce 2024-09-22T03:59:03.292 INFO:teuthology.orchestra.run.smithi052.stdout:Permissive 2024-09-22T03:59:03.292 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi052.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:59:03.292 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T03:59:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:59:03.360 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:59:03.360 DEBUG:teuthology.orchestra.run.smithi052:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:59:03.452 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:59:03.467 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-22T03:59:03.470 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-22T03:59:03.573 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:59:03.583 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:59:03.584 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T04:02:51.835 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com')] 2024-09-22T04:02:51.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T04:02:51.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T04:02:51.912 DEBUG:teuthology.orchestra.run.smithi052:> true 2024-09-22T04:02:51.995 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi052.front.sepia.ceph.com' 2024-09-22T04:02:51.995 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T04:02:52.006 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T04:02:52.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T04:02:52.007 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T04:02:52.065 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T04:02:52.080 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T04:02:52.105 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: ntpd: command not found 2024-09-22T04:02:52.116 INFO:teuthology.orchestra.run.smithi052.stdout:506 Cannot talk to daemon 2024-09-22T04:02:52.131 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T04:02:52.145 INFO:teuthology.orchestra.run.smithi052.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T04:02:52.201 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T04:02:52.205 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T04:02:52.206 INFO:teuthology.run_tasks:Running task install... 2024-09-22T04:02:52.214 DEBUG:teuthology.task.install:project ceph 2024-09-22T04:02:52.214 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-22T04:02:52.214 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-22T04:02:52.215 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T04:02:52.219 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T04:02:52.219 INFO:teuthology.task.install:extra packages: [] 2024-09-22T04:02:52.220 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-22T04:02:52.220 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T04:02:52.362 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/ 2024-09-22T04:02:52.362 INFO:teuthology.task.install.rpm:Package version is 19.3.0-5092.g62da54b4 2024-09-22T04:02:52.474 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/main/62da54b4b83a96ca9976b7e1d0c8272a564a2208/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T04:02:52.474 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:02:52.474 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T04:02:52.501 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T04:02:52.501 DEBUG:teuthology.orchestra.run.smithi052:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/62da54b4b83a96ca9976b7e1d0c8272a564a2208/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T04:02:52.567 DEBUG:teuthology.orchestra.run.smithi052:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T04:02:52.643 DEBUG:teuthology.orchestra.run.smithi052:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T04:02:52.708 INFO:teuthology.orchestra.run.smithi052.stdout:check_obsoletes = 1 2024-09-22T04:02:52.710 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum clean all 2024-09-22T04:02:52.943 INFO:teuthology.orchestra.run.smithi052.stdout:45 files removed 2024-09-22T04:02:52.971 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-radosgw 2024-09-22T04:02:53.539 INFO:teuthology.orchestra.run.smithi052.stdout:ceph packages for x86_64 327 kB/s | 88 kB 00:00 2024-09-22T04:02:53.832 INFO:teuthology.orchestra.run.smithi052.stdout:ceph noarch packages 98 kB/s | 24 kB 00:00 2024-09-22T04:02:54.107 INFO:teuthology.orchestra.run.smithi052.stdout:ceph source packages 8.8 kB/s | 2.1 kB 00:00 2024-09-22T04:02:54.716 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-22T04:02:57.917 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 20 MB 00:02 2024-09-22T04:03:03.326 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.5 MB 00:01 2024-09-22T04:03:05.043 INFO:teuthology.orchestra.run.smithi052.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 19 kB 00:00 2024-09-22T04:03:06.247 INFO:teuthology.orchestra.run.smithi052.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-22T04:03:11.953 INFO:teuthology.orchestra.run.smithi052.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-22T04:03:14.284 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 10 M 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrading: 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.8 M 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T04:03:14.298 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.1 M 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 20 M 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 25 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 808 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 558 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T04:03:14.299 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2 x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.4 M 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 45 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 148 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 159 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 314 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 302 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 99 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T04:03:14.300 INFO:teuthology.orchestra.run.smithi052.stdout:Installing weak dependencies: 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:Install 31 Packages 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:Upgrade 2 Packages 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 60 M 2024-09-22T04:03:14.301 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:03:15.238 INFO:teuthology.orchestra.run.smithi052.stdout:(1/33): ceph-base-19.3.0-5092.g62da54b4.el9.x86 13 MB/s | 6.1 MB 00:00 2024-09-22T04:03:15.296 INFO:teuthology.orchestra.run.smithi052.stdout:(2/33): ceph-selinux-19.3.0-5092.g62da54b4.el9. 431 kB/s | 25 kB 00:00 2024-09-22T04:03:15.388 INFO:teuthology.orchestra.run.smithi052.stdout:(3/33): libcephfs2-19.3.0-5092.g62da54b4.el9.x8 8.6 MB/s | 808 kB 00:00 2024-09-22T04:03:15.514 INFO:teuthology.orchestra.run.smithi052.stdout:(4/33): ceph-radosgw-19.3.0-5092.g62da54b4.el9. 14 MB/s | 10 MB 00:00 2024-09-22T04:03:15.555 INFO:teuthology.orchestra.run.smithi052.stdout:(5/33): libradosstriper1-19.3.0-5092.g62da54b4. 3.3 MB/s | 558 kB 00:00 2024-09-22T04:03:15.622 INFO:teuthology.orchestra.run.smithi052.stdout:(6/33): python3-ceph-argparse-19.3.0-5092.g62da 676 kB/s | 45 kB 00:00 2024-09-22T04:03:15.706 INFO:teuthology.orchestra.run.smithi052.stdout:(7/33): python3-ceph-common-19.3.0-5092.g62da54 1.7 MB/s | 148 kB 00:00 2024-09-22T04:03:15.781 INFO:teuthology.orchestra.run.smithi052.stdout:(8/33): librgw2-19.3.0-5092.g62da54b4.el9.x86_6 20 MB/s | 5.4 MB 00:00 2024-09-22T04:03:15.798 INFO:teuthology.orchestra.run.smithi052.stdout:(9/33): python3-cephfs-19.3.0-5092.g62da54b4.el 1.7 MB/s | 159 kB 00:00 2024-09-22T04:03:15.831 INFO:teuthology.orchestra.run.smithi052.stdout:(10/33): python3-rados-19.3.0-5092.g62da54b4.el 6.1 MB/s | 314 kB 00:00 2024-09-22T04:03:15.857 INFO:teuthology.orchestra.run.smithi052.stdout:(11/33): python3-rbd-19.3.0-5092.g62da54b4.el9. 5.0 MB/s | 302 kB 00:00 2024-09-22T04:03:15.890 INFO:teuthology.orchestra.run.smithi052.stdout:(12/33): python3-rgw-19.3.0-5092.g62da54b4.el9. 1.7 MB/s | 99 kB 00:00 2024-09-22T04:03:16.099 INFO:teuthology.orchestra.run.smithi052.stdout:(13/33): ceph-common-19.3.0-5092.g62da54b4.el9. 15 MB/s | 20 MB 00:01 2024-09-22T04:03:16.141 INFO:teuthology.orchestra.run.smithi052.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 377 kB/s | 107 kB 00:00 2024-09-22T04:03:16.166 INFO:teuthology.orchestra.run.smithi052.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 619 kB/s | 171 kB 00:00 2024-09-22T04:03:16.208 INFO:teuthology.orchestra.run.smithi052.stdout:(16/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-09-22T04:03:16.233 INFO:teuthology.orchestra.run.smithi052.stdout:(17/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 492 kB/s | 45 kB 00:00 2024-09-22T04:03:16.283 INFO:teuthology.orchestra.run.smithi052.stdout:(18/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-09-22T04:03:16.317 INFO:teuthology.orchestra.run.smithi052.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-22T04:03:16.343 INFO:teuthology.orchestra.run.smithi052.stdout:(20/33): python3-prettytable-0.7.2-27.el9.noarc 388 kB/s | 42 kB 00:00 2024-09-22T04:03:16.388 INFO:teuthology.orchestra.run.smithi052.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-22T04:03:16.393 INFO:teuthology.orchestra.run.smithi052.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-22T04:03:16.435 INFO:teuthology.orchestra.run.smithi052.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 516 kB/s | 47 kB 00:00 2024-09-22T04:03:16.477 INFO:teuthology.orchestra.run.smithi052.stdout:(24/33): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-22T04:03:16.502 INFO:teuthology.orchestra.run.smithi052.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-22T04:03:16.527 INFO:teuthology.orchestra.run.smithi052.stdout:(26/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 288 kB/s | 26 kB 00:00 2024-09-22T04:03:16.552 INFO:teuthology.orchestra.run.smithi052.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-22T04:03:16.586 INFO:teuthology.orchestra.run.smithi052.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-22T04:03:16.628 INFO:teuthology.orchestra.run.smithi052.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.2 MB/s | 839 kB 00:00 2024-09-22T04:03:16.703 INFO:teuthology.orchestra.run.smithi052.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-22T04:03:16.795 INFO:teuthology.orchestra.run.smithi052.stdout:(31/33): librados2-19.3.0-5092.g62da54b4.el9.x8 21 MB/s | 3.5 MB 00:00 2024-09-22T04:03:16.862 INFO:teuthology.orchestra.run.smithi052.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 9.4 MB/s | 4.4 MB 00:00 2024-09-22T04:03:16.971 INFO:teuthology.orchestra.run.smithi052.stdout:(33/33): librbd1-19.3.0-5092.g62da54b4.el9.x86_ 14 MB/s | 3.8 MB 00:00 2024-09-22T04:03:16.976 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:03:16.976 INFO:teuthology.orchestra.run.smithi052.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-22T04:03:17.090 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:03:17.132 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:03:17.132 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:03:17.828 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:03:17.828 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:03:19.408 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:03:19.926 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-22T04:03:20.006 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T04:03:20.099 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T04:03:20.191 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T04:03:20.216 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T04:03:20.262 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T04:03:20.308 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/35 2024-09-22T04:03:20.337 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/35 2024-09-22T04:03:20.362 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/35 2024-09-22T04:03:20.384 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T04:03:20.414 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/35 2024-09-22T04:03:20.572 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T04:03:20.610 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/35 2024-09-22T04:03:20.644 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/35 2024-09-22T04:03:20.688 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 11/35 2024-09-22T04:03:20.717 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T04:03:20.808 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 13/35 2024-09-22T04:03:20.830 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T04:03:20.920 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 14/35 2024-09-22T04:03:20.954 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : thrift-0.15.0-2.el9.x86_64 15/35 2024-09-22T04:03:21.066 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : re2-1:20211101-20.el9.x86_64 16/35 2024-09-22T04:03:21.097 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 17/35 2024-09-22T04:03:21.151 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 18/35 2024-09-22T04:03:21.179 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 19/35 2024-09-22T04:03:21.232 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 20/35 2024-09-22T04:03:21.294 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 21/35 2024-09-22T04:03:21.672 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-22T04:03:21.720 INFO:teuthology.orchestra.run.smithi052.stdout: Upgrading : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T04:03:21.812 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 23/35 2024-09-22T04:03:21.842 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 24/35 2024-09-22T04:03:21.877 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-22T04:03:22.485 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-22T04:03:22.514 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T04:03:22.577 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 27/35 2024-09-22T04:03:22.663 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 28/35 2024-09-22T04:03:22.700 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 29/35 2024-09-22T04:03:25.310 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T04:03:25.354 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T04:03:26.106 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 30/35 2024-09-22T04:03:26.140 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T04:03:26.229 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 31/35 2024-09-22T04:03:26.229 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T04:03:26.229 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T04:03:26.229 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:26.267 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /sys 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /proc 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /mnt 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /var/tmp 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /home 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /root 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:skipping the directory /tmp 2024-09-22T04:03:36.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:38.139 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T04:03:38.166 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 33/35 2024-09-22T04:03:38.167 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:03:38.167 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T04:03:38.167 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T04:03:38.167 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T04:03:38.167 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:38.169 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T04:03:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-22T04:03:38.267 INFO:teuthology.orchestra.run.smithi052.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T04:03:39.567 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 1/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 3/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 4/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 5/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_6 6/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 7/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9. 8/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x8 9/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 10/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 11/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 12/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 13/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-22T04:03:39.568 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 24/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-22T04:03:39.569 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-22T04:03:39.570 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-22T04:03:39.570 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-22T04:03:39.570 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 32/35 2024-09-22T04:03:39.570 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-22T04:03:39.570 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 34/35 2024-09-22T04:03:40.490 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-22T04:03:40.490 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:40.490 INFO:teuthology.orchestra.run.smithi052.stdout:Upgraded: 2024-09-22T04:03:40.490 INFO:teuthology.orchestra.run.smithi052.stdout: librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-radosgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-selinux-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T04:03:40.491 INFO:teuthology.orchestra.run.smithi052.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: librgw2-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-common-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T04:03:40.492 INFO:teuthology.orchestra.run.smithi052.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T04:03:40.493 INFO:teuthology.orchestra.run.smithi052.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T04:03:40.493 INFO:teuthology.orchestra.run.smithi052.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T04:03:40.493 INFO:teuthology.orchestra.run.smithi052.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T04:03:40.493 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:40.493 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:03:40.837 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-test 2024-09-22T04:03:41.438 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:30 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:03:41.579 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 81 M 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: libasan8 x86_64 14.2.1-1.2.el9 appstream 513 k 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 175 k 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T04:03:41.580 INFO:teuthology.orchestra.run.smithi052.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout:Install 9 Packages 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:03:41.581 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 83 M 2024-09-22T04:03:41.582 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 775 M 2024-09-22T04:03:41.582 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:03:42.229 INFO:teuthology.orchestra.run.smithi052.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 887 kB/s | 94 kB 00:00 2024-09-22T04:03:42.321 INFO:teuthology.orchestra.run.smithi052.stdout:(2/9): libasan8-14.2.1-1.2.el9.x86_64.rpm 5.5 MB/s | 513 kB 00:00 2024-09-22T04:03:42.355 INFO:teuthology.orchestra.run.smithi052.stdout:(3/9): libcephsqlite-19.3.0-5092.g62da54b4.el9. 755 kB/s | 175 kB 00:00 2024-09-22T04:03:42.389 INFO:teuthology.orchestra.run.smithi052.stdout:(4/9): libubsan-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 183 kB 00:00 2024-09-22T04:03:42.455 INFO:teuthology.orchestra.run.smithi052.stdout:(5/9): socat-1.7.4.1-6.el9.x86_64.rpm 4.5 MB/s | 304 kB 00:00 2024-09-22T04:03:42.506 INFO:teuthology.orchestra.run.smithi052.stdout:(6/9): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-09-22T04:03:42.531 INFO:teuthology.orchestra.run.smithi052.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 848 kB/s | 64 kB 00:00 2024-09-22T04:03:42.581 INFO:teuthology.orchestra.run.smithi052.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.6 MB/s | 122 kB 00:00 2024-09-22T04:03:44.506 INFO:teuthology.orchestra.run.smithi052.stdout:(9/9): ceph-test-19.3.0-5092.g62da54b4.el9.x86_ 34 MB/s | 81 MB 00:02 2024-09-22T04:03:44.509 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:03:44.509 INFO:teuthology.orchestra.run.smithi052.stdout:Total 28 MB/s | 83 MB 00:02 2024-09-22T04:03:44.596 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:03:44.621 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:03:44.621 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:03:45.249 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:03:45.249 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:03:46.145 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:03:46.225 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-09-22T04:03:46.275 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-09-22T04:03:46.320 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/9 2024-09-22T04:03:46.356 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-09-22T04:03:46.411 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T04:03:46.434 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T04:03:46.505 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libasan8-14.2.1-1.2.el9.x86_64 6/9 2024-09-22T04:03:46.561 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-09-22T04:03:46.603 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T04:03:58.081 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 8/9 2024-09-22T04:03:58.171 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 9/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 1/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libasan8-14.2.1-1.2.el9.x86_64 4/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 5/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 7/9 2024-09-22T04:04:04.025 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-test-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: libasan8-14.2.1-1.2.el9.x86_64 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: libcephsqlite-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-09-22T04:04:04.419 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T04:04:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T04:04:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T04:04:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T04:04:04.420 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:04.420 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:04.738 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph 2024-09-22T04:04:05.340 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:00:54 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:05.480 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout: ceph x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 6.5 k 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:04:05.483 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 2.2 M 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 1.6 M 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 292 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 5.1 M 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 70 M 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T04:04:05.484 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T04:04:05.485 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout:Install 37 Packages 2024-09-22T04:04:05.486 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:05.487 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 84 M 2024-09-22T04:04:05.487 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 896 M 2024-09-22T04:04:05.487 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:04:06.214 INFO:teuthology.orchestra.run.smithi052.stdout:(1/37): ceph-19.3.0-5092.g62da54b4.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-22T04:04:06.414 INFO:teuthology.orchestra.run.smithi052.stdout:(2/37): ceph-mds-19.3.0-5092.g62da54b4.el9.x86_ 6.7 MB/s | 2.2 MB 00:00 2024-09-22T04:04:06.456 INFO:teuthology.orchestra.run.smithi052.stdout:(3/37): ceph-mgr-19.3.0-5092.g62da54b4.el9.x86_ 4.3 MB/s | 1.6 MB 00:00 2024-09-22T04:04:06.539 INFO:teuthology.orchestra.run.smithi052.stdout:(4/37): ceph-mgr-modules-core-19.3.0-5092.g62da 3.4 MB/s | 292 kB 00:00 2024-09-22T04:04:06.748 INFO:teuthology.orchestra.run.smithi052.stdout:(5/37): ceph-mon-19.3.0-5092.g62da54b4.el9.x86_ 9.6 MB/s | 5.1 MB 00:00 2024-09-22T04:04:06.940 INFO:teuthology.orchestra.run.smithi052.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 633 kB/s | 253 kB 00:00 2024-09-22T04:04:07.057 INFO:teuthology.orchestra.run.smithi052.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 914 kB/s | 106 kB 00:00 2024-09-22T04:04:07.182 INFO:teuthology.orchestra.run.smithi052.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-09-22T04:04:07.257 INFO:teuthology.orchestra.run.smithi052.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 1.6 MB/s | 125 kB 00:00 2024-09-22T04:04:07.366 INFO:teuthology.orchestra.run.smithi052.stdout:(10/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.9 MB/s | 215 kB 00:00 2024-09-22T04:04:07.500 INFO:teuthology.orchestra.run.smithi052.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-22T04:04:07.550 INFO:teuthology.orchestra.run.smithi052.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-22T04:04:07.592 INFO:teuthology.orchestra.run.smithi052.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 835 kB/s | 35 kB 00:00 2024-09-22T04:04:07.642 INFO:teuthology.orchestra.run.smithi052.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-22T04:04:07.684 INFO:teuthology.orchestra.run.smithi052.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-22T04:04:07.726 INFO:teuthology.orchestra.run.smithi052.stdout:(16/37): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-09-22T04:04:08.368 INFO:teuthology.orchestra.run.smithi052.stdout:(17/37): ceph-osd-19.3.0-5092.g62da54b4.el9.x86 36 MB/s | 70 MB 00:01 2024-09-22T04:04:08.427 INFO:teuthology.orchestra.run.smithi052.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 32 kB/s | 22 kB 00:00 2024-09-22T04:04:08.461 INFO:teuthology.orchestra.run.smithi052.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-09-22T04:04:08.494 INFO:teuthology.orchestra.run.smithi052.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 653 kB/s | 43 kB 00:00 2024-09-22T04:04:08.519 INFO:teuthology.orchestra.run.smithi052.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.0 MB/s | 172 kB 00:00 2024-09-22T04:04:08.561 INFO:teuthology.orchestra.run.smithi052.stdout:(22/37): python3-cryptography-36.0.1-4.el9.x86_ 693 kB/s | 1.2 MB 00:01 2024-09-22T04:04:08.586 INFO:teuthology.orchestra.run.smithi052.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-09-22T04:04:08.620 INFO:teuthology.orchestra.run.smithi052.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 943 kB/s | 23 kB 00:00 2024-09-22T04:04:08.661 INFO:teuthology.orchestra.run.smithi052.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 494 kB/s | 19 kB 00:00 2024-09-22T04:04:08.679 INFO:teuthology.orchestra.run.smithi052.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 150 kB/s | 18 kB 00:00 2024-09-22T04:04:08.695 INFO:teuthology.orchestra.run.smithi052.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 586 kB/s | 20 kB 00:00 2024-09-22T04:04:08.721 INFO:teuthology.orchestra.run.smithi052.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-22T04:04:08.737 INFO:teuthology.orchestra.run.smithi052.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 791 kB/s | 46 kB 00:00 2024-09-22T04:04:08.763 INFO:teuthology.orchestra.run.smithi052.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-22T04:04:08.779 INFO:teuthology.orchestra.run.smithi052.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-09-22T04:04:08.805 INFO:teuthology.orchestra.run.smithi052.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-22T04:04:08.830 INFO:teuthology.orchestra.run.smithi052.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-22T04:04:08.864 INFO:teuthology.orchestra.run.smithi052.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-22T04:04:08.889 INFO:teuthology.orchestra.run.smithi052.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 809 kB/s | 20 kB 00:00 2024-09-22T04:04:08.914 INFO:teuthology.orchestra.run.smithi052.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-22T04:04:08.939 INFO:teuthology.orchestra.run.smithi052.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-22T04:04:08.944 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:04:08.944 INFO:teuthology.orchestra.run.smithi052.stdout:Total 24 MB/s | 84 MB 00:03 2024-09-22T04:04:09.098 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:04:09.145 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:04:09.145 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:04:09.982 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:04:09.982 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:04:11.672 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:04:11.734 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-22T04:04:11.767 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-22T04:04:11.794 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-22T04:04:11.869 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-22T04:04:11.919 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-22T04:04:11.952 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-22T04:04:12.048 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-22T04:04:12.125 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-22T04:04:12.171 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-22T04:04:12.206 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-22T04:04:12.235 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-22T04:04:12.270 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-22T04:04:12.303 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-22T04:04:12.351 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-22T04:04:12.395 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-22T04:04:12.428 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-22T04:04:12.486 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-22T04:04:12.525 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-22T04:04:12.602 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-22T04:04:12.753 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-22T04:04:12.836 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-22T04:04:12.888 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-22T04:04:12.934 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-22T04:04:12.991 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-22T04:04:13.055 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-22T04:04:13.250 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-22T04:04:13.295 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-22T04:04:13.376 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-22T04:04:13.465 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-22T04:04:13.507 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-22T04:04:13.646 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-22T04:04:13.856 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 32/37 2024-09-22T04:04:13.887 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 33/37 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T04:04:13.914 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:25.087 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T04:04:25.116 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 34/37 2024-09-22T04:04:25.116 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:04:25.117 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T04:04:25.117 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T04:04:25.117 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T04:04:25.117 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:25.688 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 35/37 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T04:04:25.713 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:25.964 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T04:04:25.988 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 36/37 2024-09-22T04:04:25.988 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:04:25.988 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T04:04:25.989 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T04:04:25.989 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T04:04:25.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:26.056 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T04:04:32.675 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 37/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 1/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 3/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 4/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 5/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9. 6/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-22T04:04:32.676 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-22T04:04:32.677 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-22T04:04:32.678 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-22T04:04:32.679 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-22T04:04:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-22T04:04:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:33.639 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:04:33.639 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mds-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-modules-core-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mon-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-osd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T04:04:33.640 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T04:04:33.641 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:33.642 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:34.025 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-base 2024-09-22T04:04:34.630 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:23 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:34.723 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-base-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:04:34.768 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:34.769 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:04:34.769 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:34.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install cephadm 2024-09-22T04:04:35.423 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:24 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:35.559 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 784 k 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:04:35.560 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:35.561 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 784 k 2024-09-22T04:04:35.561 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 791 k 2024-09-22T04:04:35.561 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:04:35.814 INFO:teuthology.orchestra.run.smithi052.stdout:cephadm-19.3.0-5092.g62da54b4.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-22T04:04:35.815 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:04:35.815 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-22T04:04:35.815 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:04:35.821 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:04:35.821 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:04:35.827 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:04:35.828 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:04:35.902 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:04:36.337 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:04:36.430 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:04:36.808 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout: cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:37.058 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:37.285 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T04:04:37.915 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:26 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:38.050 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 158 k 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:04:38.051 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:38.052 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 158 k 2024-09-22T04:04:38.052 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 644 k 2024-09-22T04:04:38.052 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:04:38.270 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-immutable-object-cache-19.3.0-5092.g62da54 725 kB/s | 158 kB 00:00 2024-09-22T04:04:38.271 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:04:38.271 INFO:teuthology.orchestra.run.smithi052.stdout:Total 720 kB/s | 158 kB 00:00 2024-09-22T04:04:38.271 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:04:38.282 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:04:38.282 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:04:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:04:38.344 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:04:38.494 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:04:38.554 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T04:04:38.580 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T04:04:38.580 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:04:38.581 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T04:04:38.581 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4. 1/1 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-immutable-object-cache-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:39.563 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:39.737 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr 2024-09-22T04:04:40.341 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:29 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:40.434 INFO:teuthology.orchestra.run.smithi052.stdout:Package ceph-mgr-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:04:40.478 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:40.479 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:04:40.479 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:40.542 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T04:04:41.144 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:30 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:41.282 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:41.283 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 14 M 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 34 k 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 16 k 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T04:04:41.284 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout:Install 11 Packages 2024-09-22T04:04:41.285 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:41.286 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 18 M 2024-09-22T04:04:41.286 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 156 M 2024-09-22T04:04:41.286 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:04:41.898 INFO:teuthology.orchestra.run.smithi052.stdout:(1/11): ceph-grafana-dashboards-19.3.0-5092.g62 231 kB/s | 34 kB 00:00 2024-09-22T04:04:41.914 INFO:teuthology.orchestra.run.smithi052.stdout:(2/11): ceph-prometheus-alerts-19.3.0-5092.g62d 99 kB/s | 16 kB 00:00 2024-09-22T04:04:42.390 INFO:teuthology.orchestra.run.smithi052.stdout:(3/11): ceph-mgr-dashboard-19.3.0-5092.g62da54b 22 MB/s | 14 MB 00:00 2024-09-22T04:04:42.450 INFO:teuthology.orchestra.run.smithi052.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 9.3 MB/s | 548 kB 00:00 2024-09-22T04:04:42.465 INFO:teuthology.orchestra.run.smithi052.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-09-22T04:04:42.541 INFO:teuthology.orchestra.run.smithi052.stdout:(6/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 27 MB/s | 2.0 MB 00:00 2024-09-22T04:04:42.783 INFO:teuthology.orchestra.run.smithi052.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 597 kB/s | 144 kB 00:00 2024-09-22T04:04:42.799 INFO:teuthology.orchestra.run.smithi052.stdout:(8/11): python3-protobuf-3.14.0-13.el9.noarch.r 299 kB/s | 269 kB 00:00 2024-09-22T04:04:42.825 INFO:teuthology.orchestra.run.smithi052.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 737 kB/s | 31 kB 00:00 2024-09-22T04:04:42.866 INFO:teuthology.orchestra.run.smithi052.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-22T04:04:42.942 INFO:teuthology.orchestra.run.smithi052.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 841 kB/s | 863 kB 00:01 2024-09-22T04:04:42.948 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:04:42.948 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 18 MB 00:01 2024-09-22T04:04:42.982 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:04:43.004 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:04:43.005 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:04:43.268 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:04:43.269 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:04:43.869 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:04:43.950 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T04:04:43.987 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T04:04:44.091 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T04:04:44.236 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T04:04:44.282 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T04:04:44.378 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T04:04:44.426 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T04:04:44.453 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T04:04:44.486 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 9/11 2024-09-22T04:04:47.921 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 10/11 2024-09-22T04:04:47.973 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 11/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el 1/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noa 2/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9 3/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T04:04:49.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-grafana-dashboards-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-prometheus-alerts-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T04:04:50.178 INFO:teuthology.orchestra.run.smithi052.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:50.179 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:04:50.440 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T04:04:51.039 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:40 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:04:51.177 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:04:51.178 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 7.4 M 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T04:04:51.179 INFO:teuthology.orchestra.run.smithi052.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout:======================================================================================= 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout:Install 12 Packages 2024-09-22T04:04:51.180 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:04:51.181 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 38 M 2024-09-22T04:04:51.181 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 212 M 2024-09-22T04:04:51.181 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:04:51.729 INFO:teuthology.orchestra.run.smithi052.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.3 MB/s | 191 kB 00:00 2024-09-22T04:04:51.755 INFO:teuthology.orchestra.run.smithi052.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.6 MB/s | 801 kB 00:00 2024-09-22T04:04:51.805 INFO:teuthology.orchestra.run.smithi052.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-09-22T04:04:51.880 INFO:teuthology.orchestra.run.smithi052.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 24 MB/s | 3.0 MB 00:00 2024-09-22T04:04:51.906 INFO:teuthology.orchestra.run.smithi052.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 181 kB/s | 18 kB 00:00 2024-09-22T04:04:51.939 INFO:teuthology.orchestra.run.smithi052.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-22T04:04:51.998 INFO:teuthology.orchestra.run.smithi052.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-09-22T04:04:52.098 INFO:teuthology.orchestra.run.smithi052.stdout:(8/12): ceph-mgr-diskprediction-local-19.3.0-50 14 MB/s | 7.4 MB 00:00 2024-09-22T04:04:52.173 INFO:teuthology.orchestra.run.smithi052.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2024-09-22T04:04:52.248 INFO:teuthology.orchestra.run.smithi052.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-09-22T04:04:52.299 INFO:teuthology.orchestra.run.smithi052.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.2 MB/s | 250 kB 00:00 2024-09-22T04:04:52.532 INFO:teuthology.orchestra.run.smithi052.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-09-22T04:04:52.533 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:04:52.533 INFO:teuthology.orchestra.run.smithi052.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-22T04:04:52.897 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:04:52.924 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:04:52.925 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:04:53.383 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:04:53.383 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:04:54.175 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:04:54.254 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T04:04:54.289 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T04:04:54.394 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T04:04:54.432 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T04:04:54.625 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T04:04:54.655 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T04:04:54.735 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T04:04:55.423 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T04:04:55.540 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T04:04:56.859 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T04:04:57.940 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T04:04:57.970 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T04:05:00.530 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 12/12 2024-09-22T04:05:00.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da5 1/12 2024-09-22T04:05:00.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T04:05:00.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T04:05:00.530 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T04:05:00.531 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T04:05:00.945 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T04:05:00.945 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:00.945 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:00.945 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:05:00.945 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:00.946 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:01.320 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-rook 2024-09-22T04:05:01.931 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:50 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:02.070 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 49 k 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:05:02.071 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:02.072 INFO:teuthology.orchestra.run.smithi052.stdout:Install 13 Packages 2024-09-22T04:05:02.073 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:02.073 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.2 M 2024-09-22T04:05:02.073 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 26 M 2024-09-22T04:05:02.073 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:02.378 INFO:teuthology.orchestra.run.smithi052.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 231 kB/s | 26 kB 00:00 2024-09-22T04:05:02.403 INFO:teuthology.orchestra.run.smithi052.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 19 kB 00:00 2024-09-22T04:05:02.428 INFO:teuthology.orchestra.run.smithi052.stdout:(3/13): ceph-mgr-rook-19.3.0-5092.g62da54b4.el9 305 kB/s | 49 kB 00:00 2024-09-22T04:05:02.520 INFO:teuthology.orchestra.run.smithi052.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-22T04:05:02.545 INFO:teuthology.orchestra.run.smithi052.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-22T04:05:02.596 INFO:teuthology.orchestra.run.smithi052.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.1 MB/s | 54 kB 00:00 2024-09-22T04:05:02.621 INFO:teuthology.orchestra.run.smithi052.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 429 kB/s | 32 kB 00:00 2024-09-22T04:05:02.654 INFO:teuthology.orchestra.run.smithi052.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-22T04:05:02.688 INFO:teuthology.orchestra.run.smithi052.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-22T04:05:02.713 INFO:teuthology.orchestra.run.smithi052.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.5 MB/s | 220 kB 00:00 2024-09-22T04:05:02.738 INFO:teuthology.orchestra.run.smithi052.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-22T04:05:02.764 INFO:teuthology.orchestra.run.smithi052.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-22T04:05:02.822 INFO:teuthology.orchestra.run.smithi052.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.2 MB/s | 1.0 MB 00:00 2024-09-22T04:05:02.825 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:02.825 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.9 MB/s | 2.2 MB 00:00 2024-09-22T04:05:02.893 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:02.910 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:02.910 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:03.007 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:03.008 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:03.534 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:03.718 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T04:05:03.775 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T04:05:03.830 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T04:05:03.866 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T04:05:03.900 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T04:05:04.008 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T04:05:04.135 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T04:05:04.186 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T04:05:04.942 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T04:05:04.998 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T04:05:05.031 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T04:05:05.081 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T04:05:05.109 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T04:05:05.910 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 13/13 2024-09-22T04:05:05.910 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 1/13 2024-09-22T04:05:05.910 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T04:05:05.910 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T04:05:05.911 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T04:05:06.236 INFO:teuthology.orchestra.run.smithi052.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:06.237 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:06.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T04:05:07.070 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:01:56 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:07.209 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 154 k 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T04:05:07.210 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout:Install 6 Packages 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 7.0 M 2024-09-22T04:05:07.211 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 33 M 2024-09-22T04:05:07.212 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:07.424 INFO:teuthology.orchestra.run.smithi052.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-22T04:05:07.507 INFO:teuthology.orchestra.run.smithi052.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-22T04:05:07.532 INFO:teuthology.orchestra.run.smithi052.stdout:(3/6): ceph-mgr-cephadm-19.3.0-5092.g62da54b4.e 587 kB/s | 154 kB 00:00 2024-09-22T04:05:07.558 INFO:teuthology.orchestra.run.smithi052.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-22T04:05:07.591 INFO:teuthology.orchestra.run.smithi052.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-22T04:05:08.791 INFO:teuthology.orchestra.run.smithi052.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.9 MB/s | 6.0 MB 00:01 2024-09-22T04:05:08.794 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:08.795 INFO:teuthology.orchestra.run.smithi052.stdout:Total 4.4 MB/s | 7.0 MB 00:01 2024-09-22T04:05:08.871 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:08.884 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:08.884 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:08.992 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:08.992 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:09.310 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:09.483 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T04:05:09.545 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T04:05:10.121 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T04:05:10.208 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T04:05:10.283 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T04:05:10.314 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 6/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 1/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T04:05:11.089 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-cephadm-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T04:05:11.358 INFO:teuthology.orchestra.run.smithi052.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T04:05:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T04:05:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T04:05:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T04:05:11.359 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:11.359 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:11.606 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-fuse 2024-09-22T04:05:12.212 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:01 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:12.351 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 883 k 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout:Install 2 Packages 2024-09-22T04:05:12.352 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:12.353 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 963 k 2024-09-22T04:05:12.353 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 4.1 M 2024-09-22T04:05:12.353 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:12.518 INFO:teuthology.orchestra.run.smithi052.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 561 kB/s | 80 kB 00:00 2024-09-22T04:05:12.660 INFO:teuthology.orchestra.run.smithi052.stdout:(2/2): ceph-fuse-19.3.0-5092.g62da54b4.el9.x86_ 3.0 MB/s | 883 kB 00:00 2024-09-22T04:05:12.661 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:12.661 INFO:teuthology.orchestra.run.smithi052.stdout:Total 3.1 MB/s | 963 kB 00:00 2024-09-22T04:05:12.673 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:12.699 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:12.699 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:12.785 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:12.785 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:12.993 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:13.143 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T04:05:13.222 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T04:05:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2/2 2024-09-22T04:05:13.944 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/2 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:14.224 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:14.483 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install ceph-volume 2024-09-22T04:05:15.099 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:04 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:15.238 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume noarch 2:19.3.0-5092.g62da54b4.el9 ceph-noarch 285 k 2024-09-22T04:05:15.239 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 285 k 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 1.3 M 2024-09-22T04:05:15.240 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:15.464 INFO:teuthology.orchestra.run.smithi052.stdout:ceph-volume-19.3.0-5092.g62da54b4.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-22T04:05:15.464 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:15.465 INFO:teuthology.orchestra.run.smithi052.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T04:05:15.465 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:15.474 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:15.474 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:15.509 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:15.509 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:15.739 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:15.807 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:05:15.824 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:05:15.824 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:05:15.824 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T04:05:15.824 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 1/1 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-volume-2:19.3.0-5092.g62da54b4.el9.noarch 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:16.857 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:17.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados-devel 2024-09-22T04:05:17.752 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:06 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:17.891 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 125 k 2024-09-22T04:05:17.892 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 125 k 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 445 k 2024-09-22T04:05:17.893 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:18.077 INFO:teuthology.orchestra.run.smithi052.stdout:librados-devel-19.3.0-5092.g62da54b4.el9.x86_64 680 kB/s | 125 kB 00:00 2024-09-22T04:05:18.077 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:18.078 INFO:teuthology.orchestra.run.smithi052.stdout:Total 674 kB/s | 125 kB 00:00 2024-09-22T04:05:18.078 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:18.084 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:18.084 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:18.151 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:18.152 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:18.305 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:18.427 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:18.878 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout: librados-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:19.152 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:19.396 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs2 2024-09-22T04:05:20.007 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:20.100 INFO:teuthology.orchestra.run.smithi052.stdout:Package libcephfs2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:20.146 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:20.147 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:20.147 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:20.209 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install libcephfs-devel 2024-09-22T04:05:20.820 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:09 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:20.962 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:20.962 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:20.962 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-22T04:05:20.962 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 32 k 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:20.963 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 32 k 2024-09-22T04:05:20.964 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 142 k 2024-09-22T04:05:20.964 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:21.121 INFO:teuthology.orchestra.run.smithi052.stdout:libcephfs-devel-19.3.0-5092.g62da54b4.el9.x86_6 202 kB/s | 32 kB 00:00 2024-09-22T04:05:21.121 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:21.121 INFO:teuthology.orchestra.run.smithi052.stdout:Total 200 kB/s | 32 kB 00:00 2024-09-22T04:05:21.122 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:21.125 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:21.125 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:21.151 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:21.152 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:21.249 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:21.363 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:21.757 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:21.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:21.980 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:05:21.981 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:21.981 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:22.153 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librados2 2024-09-22T04:05:22.763 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:11 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:22.857 INFO:teuthology.orchestra.run.smithi052.stdout:Package librados2-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:22.903 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:22.904 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:22.904 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:22.968 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install librbd1 2024-09-22T04:05:23.577 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:12 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:23.670 INFO:teuthology.orchestra.run.smithi052.stdout:Package librbd1-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:23.717 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:23.718 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:23.718 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:23.787 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rados 2024-09-22T04:05:24.388 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:13 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:24.481 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rados-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:24.527 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:24.528 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:24.528 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:24.591 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rgw 2024-09-22T04:05:25.232 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:14 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:25.324 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rgw-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:25.370 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:25.372 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:25.372 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:25.445 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-cephfs 2024-09-22T04:05:26.047 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:15 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:26.139 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-cephfs-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:26.185 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:26.186 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:26.186 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:26.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-rbd 2024-09-22T04:05:26.857 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:15 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:26.949 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-rbd-2:19.3.0-5092.g62da54b4.el9.x86_64 is already installed. 2024-09-22T04:05:26.995 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:26.996 INFO:teuthology.orchestra.run.smithi052.stdout:Nothing to do. 2024-09-22T04:05:26.996 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:27.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-fuse 2024-09-22T04:05:27.676 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:16 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:27.814 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 91 k 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:27.815 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:27.816 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:27.816 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 91 k 2024-09-22T04:05:27.816 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 321 k 2024-09-22T04:05:27.816 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:28.015 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-fuse-19.3.0-5092.g62da54b4.el9.x86_64.rpm 457 kB/s | 91 kB 00:00 2024-09-22T04:05:28.016 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:28.016 INFO:teuthology.orchestra.run.smithi052.stdout:Total 454 kB/s | 91 kB 00:00 2024-09-22T04:05:28.016 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:28.026 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:28.026 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:28.093 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:28.093 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:28.240 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:28.349 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:28.770 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-fuse-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:29.067 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:29.323 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-mirror 2024-09-22T04:05:29.939 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:18 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:30.079 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:30.079 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repo Size 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 3.5 M 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 3.5 M 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 30 M 2024-09-22T04:05:30.080 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:30.410 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-mirror-19.3.0-5092.g62da54b4.el9.x86_64.rpm 11 MB/s | 3.5 MB 00:00 2024-09-22T04:05:30.411 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:30.411 INFO:teuthology.orchestra.run.smithi052.stdout:Total 11 MB/s | 3.5 MB 00:00 2024-09-22T04:05:30.411 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:30.424 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:30.424 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:30.512 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:30.512 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:31.129 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:31.194 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:31.222 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:31.222 INFO:teuthology.orchestra.run.smithi052.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T04:05:31.222 INFO:teuthology.orchestra.run.smithi052.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T04:05:31.222 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T04:05:31.222 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T04:05:31.223 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-mirror-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:32.513 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:32.755 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install rbd-nbd 2024-09-22T04:05:33.371 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:05:33.511 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:05:33.511 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:33.511 INFO:teuthology.orchestra.run.smithi052.stdout: Package Architecture Version Repository Size 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 176 k 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Install 1 Package 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 176 k 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 685 k 2024-09-22T04:05:33.512 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:05:33.739 INFO:teuthology.orchestra.run.smithi052.stdout:rbd-nbd-19.3.0-5092.g62da54b4.el9.x86_64.rpm 781 kB/s | 176 kB 00:00 2024-09-22T04:05:33.739 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:05:33.739 INFO:teuthology.orchestra.run.smithi052.stdout:Total 775 kB/s | 176 kB 00:00 2024-09-22T04:05:33.739 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:05:33.751 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:05:33.751 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:05:33.819 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:05:33.819 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:05:34.018 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:05:34.123 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:34.572 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 1/1 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout: rbd-nbd-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:34.831 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:05:35.094 DEBUG:teuthology.parallel:result is None 2024-09-22T04:05:35.094 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-22T04:05:35.242 DEBUG:teuthology.orchestra.run.smithi052:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T04:05:35.264 INFO:teuthology.orchestra.run.smithi052.stdout:19.3.0-5092.g62da54b4.el9 2024-09-22T04:05:35.264 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5092.g62da54b4.el9 2024-09-22T04:05:35.264 INFO:teuthology.task.install:The correct ceph version 19.3.0-5092.g62da54b4 is installed. 2024-09-22T04:05:35.266 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T04:05:35.266 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:35.266 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T04:05:35.331 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T04:05:35.331 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:35.331 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T04:05:35.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T04:05:35.463 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T04:05:35.463 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:35.463 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T04:05:35.529 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T04:05:35.595 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T04:05:35.595 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:35.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T04:05:35.661 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T04:05:35.727 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T04:05:35.803 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T04:05:35.803 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2024-09-22T04:05:35.865 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T04:05:35.865 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T04:05:35.890 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T04:05:35.890 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T04:05:35.963 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T04:05:35.963 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T04:05:35.963 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406', 'branch': 'main', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914406', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 2950, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'targets': {'smithi052.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBaikvqvxRnYyPC5OuJ2Mk9oe2O/HKR+HJsUR4+aXu2IFjGYrFUPM3qTgJ84PjjOBC1EdxKarea92BkIAcvbp6g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_20:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T04:05:35.964 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T04:05:36.013 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T04:05:36.079 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:36.079 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T04:05:36.136 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T04:05:36.136 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-22 04:05:31.484705052 +0000 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-22 04:02:32.553222718 +0000 2024-09-22T04:05:36.192 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-22 04:02:32.553222718 +0000 2024-09-22T04:05:36.193 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-22 04:02:32.553222718 +0000 2024-09-22T04:05:36.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T04:05:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-22T04:05:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-22T04:05:36.257 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 9.7241e-05 s, 5.3 MB/s 2024-09-22T04:05:36.258 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T04:05:36.315 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-22 04:05:31.484705052 +0000 2024-09-22T04:05:36.371 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-22 04:02:32.898218002 +0000 2024-09-22T04:05:36.372 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-22 04:02:32.898218002 +0000 2024-09-22T04:05:36.372 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-22 04:02:32.898218002 +0000 2024-09-22T04:05:36.372 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T04:05:36.435 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-22T04:05:36.435 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-22T04:05:36.435 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000103909 s, 4.9 MB/s 2024-09-22T04:05:36.437 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T04:05:36.494 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T04:05:36.550 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-22 04:05:31.484705052 +0000 2024-09-22T04:05:36.551 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-22 04:02:33.220213601 +0000 2024-09-22T04:05:36.551 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-22 04:02:33.220213601 +0000 2024-09-22T04:05:36.551 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-22 04:02:33.220213601 +0000 2024-09-22T04:05:36.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T04:05:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-22T04:05:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-22T04:05:36.614 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.00010026 s, 5.1 MB/s 2024-09-22T04:05:36.615 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T04:05:36.672 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2024-09-22 04:05:31.485705038 +0000 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2024-09-22 04:02:33.546209146 +0000 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2024-09-22 04:02:33.546209146 +0000 2024-09-22T04:05:36.728 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: 2024-09-22 04:02:33.546209146 +0000 2024-09-22T04:05:36.729 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T04:05:36.792 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2024-09-22T04:05:36.793 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2024-09-22T04:05:36.793 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000100955 s, 5.1 MB/s 2024-09-22T04:05:36.794 DEBUG:teuthology.orchestra.run.smithi052:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T04:05:36.851 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T04:05:36.851 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T04:05:36.851 INFO:tasks.ceph:Generating config... 2024-09-22T04:05:36.852 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T04:05:36.852 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T04:05:36.852 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore backref = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore cache = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore device = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore epm = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore journal = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore lba = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore odata = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore ompap = 20 2024-09-22T04:05:36.853 INFO:tasks.ceph:[osd] debug seastore onode = 20 2024-09-22T04:05:36.854 INFO:tasks.ceph:[osd] debug seastore t = 20 2024-09-22T04:05:36.854 INFO:tasks.ceph:[osd] debug seastore tm = 20 2024-09-22T04:05:36.854 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T04:05:36.854 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T04:05:36.854 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T04:05:36.855 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T04:05:36.856 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T04:05:36.857 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T04:05:36.858 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T04:05:36.859 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T04:05:36.859 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T04:05:36.859 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T04:05:36.859 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T04:05:36.859 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T04:05:36.859 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T04:05:36.859 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T04:05:36.859 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T04:05:36.859 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T04:05:36.859 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T04:05:36.859 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T04:05:36.953 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T04:05:36.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T04:05:37.008 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T04:05:37.034 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.52')] 2024-09-22T04:05:37.034 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.52', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore ompap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T04:05:37.035 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:37.035 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T04:05:37.090 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.52 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:37.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:37.182+0000 7f49cb6dbf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:37.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:37.182+0000 7f49cb6dbf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:37.184 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:37.182+0000 7f49cb6dbf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid c12a7beb-40fd-4d8a-9fc4-1739c65f3b64 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:setting min_mon_release = quincy 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:fsid c12a7beb-40fd-4d8a-9fc4-1739c65f3b64 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2024-09-22T04:05:37.184470+0000 2024-09-22T04:05:37.185 INFO:teuthology.orchestra.run.smithi052.stdout:created 2024-09-22T04:05:37.184470+0000 2024-09-22T04:05:37.186 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 17 (quincy) 2024-09-22T04:05:37.186 INFO:teuthology.orchestra.run.smithi052.stdout:election_strategy: 1 2024-09-22T04:05:37.186 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2024-09-22T04:05:37.186 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T04:05:37.187 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T04:05:37.202 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c12a7beb-40fd-4d8a-9fc4-1739c65f3b64... 2024-09-22T04:05:37.204 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = none 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2024-09-22T04:05:37.289 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2024-09-22T04:05:37.290 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2024-09-22T04:05:37.291 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.52 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: auth client required = none 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: auth cluster required = none 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: auth service required = none 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug asok = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug bdev = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluefs = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug bluestore = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug buffer = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug civetweb = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug client = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug compressor = 0/0 2024-09-22T04:05:37.292 INFO:teuthology.orchestra.run.smithi052.stdout: debug context = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug crush = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug crypto = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug dpdk = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug eventtrace = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug filer = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug finisher = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug fuse = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug heartbeatmap = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug javaclient = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug journaler = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug kstore = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug leveldb = 0/0 2024-09-22T04:05:37.293 INFO:teuthology.orchestra.run.smithi052.stdout: debug lockdep = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds balancer = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds locker = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds log = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds log expire = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mds migrator = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug memdb = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgrc = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug objclass = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug objectcacher = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug objecter = 0/0 2024-09-22T04:05:37.294 INFO:teuthology.orchestra.run.smithi052.stdout: debug optracker = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug perfcounter = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rados = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd mirror = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rbd replay = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug refs = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw lifecycle = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw notification = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw sync = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug rocksdb = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug striper = 0/0 2024-09-22T04:05:37.295 INFO:teuthology.orchestra.run.smithi052.stdout: debug throttle = 0/0 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: debug timer = 0/0 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: debug tp = 0/0 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms client mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms cluster mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms mon client mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms mon cluster mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms mon service mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: ms service mode = crc 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = c12a7beb-40fd-4d8a-9fc4-1739c65f3b64 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2024-09-22T04:05:37.296 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd mclock profile = high_recovery_ops 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug trim objects = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2024-09-22T04:05:37.297 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: debug monc = 20 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T04:05:37.298 INFO:teuthology.orchestra.run.smithi052.stdout: crimson osd obc lru size = 10 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug filestore = 0/0 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug journal = 0/0 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 0/0 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 0/0 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore backref = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore cache = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore cleaner = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore device = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore epm = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore fixedkv tree = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore journal = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore lba = 20 2024-09-22T04:05:37.299 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore odata = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore ompap = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore onode = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore t = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug seastore tm = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: osd objectstore = seastore 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: osd shutdown pgref assert = True 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2024-09-22T04:05:37.300 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 0/0 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 0/0 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 0/0 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T04:05:37.301 INFO:teuthology.orchestra.run.smithi052.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: # 1m isn't quite enough 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: mon_down_mkfs_grace = 2m 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_filestore_osds = false 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default crimson = True 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2024-09-22T04:05:37.302 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: crimson seastar cpu cores = 0-2 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: crimson seastar cpu cores = 3-5 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: crimson seastar cpu cores = 0-2 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.3] 2024-09-22T04:05:37.303 INFO:teuthology.orchestra.run.smithi052.stdout: crimson seastar cpu cores = 3-5 2024-09-22T04:05:37.304 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T04:05:37.304 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T04:05:37.372 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T04:05:37.373 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:37.373 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T04:05:37.388 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:37.388 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T04:05:37.444 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:05:37.444 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:37.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T04:05:37.444 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T04:05:37.520 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:37.521 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:37.577 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T04:05:37.577 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T04:05:37.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T04:05:37.679 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T04:05:37.681 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T04:05:37.682 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T04:05:37.682 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T04:05:37.732 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T04:05:37.745 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T04:05:37.745 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T04:05:37.746 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T04:05:37.771 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T04:05:37.771 INFO:tasks.ceph:role: osd.0 2024-09-22T04:05:37.771 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:05:37.771 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T04:05:37.836 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T04:05:37.836 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T04:05:37.836 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T04:05:37.836 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T04:05:37.836 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T04:05:37.837 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-22T04:05:37.837 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T04:05:37.837 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T04:05:37.837 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T04:05:37.837 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T04:05:37.869 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-22T04:05:37.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-22T04:05:37.871 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T04:05:37.940 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T04:05:38.010 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T04:05:38.077 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T04:05:38.078 INFO:tasks.ceph:role: osd.1 2024-09-22T04:05:38.078 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:05:38.078 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T04:05:38.143 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T04:05:38.143 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T04:05:38.143 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T04:05:38.143 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T04:05:38.144 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T04:05:38.175 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-22T04:05:38.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-22T04:05:38.177 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T04:05:38.247 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T04:05:38.319 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T04:05:38.385 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T04:05:38.385 INFO:tasks.ceph:role: osd.2 2024-09-22T04:05:38.385 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:05:38.386 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T04:05:38.450 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T04:05:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T04:05:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T04:05:38.450 INFO:teuthology.orchestra.run.smithi052.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T04:05:38.451 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T04:05:38.482 INFO:teuthology.orchestra.run.smithi052.stdout:Discarding blocks...Done. 2024-09-22T04:05:38.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2024-09-22T04:05:38.484 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T04:05:38.553 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T04:05:38.623 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:38.926 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:38.927 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:38,927 seastar - Reactor backend: linux-aio 2024-09-22T04:05:38.942 INFO:teuthology.orchestra.run.smithi052.stderr:==28196==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:39.083 INFO:teuthology.orchestra.run.smithi052.stderr:DEBUG 2024-09-22 04:05:39,083 [shard 0:main] seastar - reactor::drain 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:39.628 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:39.686 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:39.690 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:39,690 seastar - Reactor backend: linux-aio 2024-09-22T04:05:39.701 INFO:teuthology.orchestra.run.smithi052.stderr:==28194==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:39.765 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x396c22a 0x4585f9c 0x462290d 0x5fbf0a1 0x5fbf2d2 0x5f9c36b 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:39.765 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:39.941 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8c12a 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x968cf8c 0x968d328 0x968d4ad 0x968d54f 0x968d668 0xbe289a9 0xbe28bb0 0xbe28c86 0xbe28dc1 0xbe291b5 0xbe28578 0xbe28658 0xbe2876c 0xbe2842f 0x94eb330 0x95d0372 0x9307833 0x9307a0d 0x9307b48 0x9307c02 0x9307cdc 0x9308228 0x9308500 0x93085b5 0x957fcfc 0x957ff61 0x95ae198 0x96b1946 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:39.941 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x390ff 0x4ab1c 0x3ac7302 0x3b8c153 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xff4d8 0x425e1 0x3ef7b 0xfd531 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb837196 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xba13cb0 0xba13f4c 0xba14113 0xba14387 0xba14490 0xba1460c 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.000 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.058 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x95c7636 0x9826b49 0x9a94970 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.088 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.088 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb836d8a 0xb837301 0xb83770c 0xb8379cb 0xba13cb0 0xba13f4c 0xba14113 0xba14387 0xba14490 0xba1460c 0xba14db8 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.088 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.088 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x15a45f 0x425e1 0x3ef7b 0xfd531 0x95c6055 0x95c6cac 0x95c7175 0x9adca6b 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:40.088 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.116 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5c38c55 0x76be4f1 0x76be71d 0xba0e265 0xb9b5385 0xb9b58b4 0xb9bb877 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:40.117 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.159 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab92 0xb9cc4a3 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.159 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.159 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x8b571 0x10d76c 0x10af7c 0x446e6 0xfb4fb 0x3767e9b 0xb9ffe85 0xb9bfd6e 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.159 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.216 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133d18 0xdd332 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xba15051 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.216 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a1cf 0xfb524 0x3767e9b 0xb9ffe85 0xba0b7a4 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.216 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.216 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.256 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16cd4 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:40.256 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.353 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x9e16cd4 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.353 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:40.354 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525df6 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:40.354 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:41.840 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:41.841 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:41.841 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:41.841 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:41.841 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:41.868 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T04:05:41.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:42.190 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:42.191 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:42,191 seastar - Reactor backend: linux-aio 2024-09-22T04:05:42.197 INFO:teuthology.orchestra.run.smithi052.stderr:==28243==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:42.305 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156dc 0xdd312 0x4b7ec 0x5fc061d 0x5fc09aa 0x601bbd4 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:42.306 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:42.338 INFO:teuthology.orchestra.run.smithi052.stderr:DEBUG 2024-09-22 04:05:42,338 [shard 0:main] seastar - reactor::drain 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:42.872 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:42.931 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:42.935 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:42,935 seastar - Reactor backend: linux-aio 2024-09-22T04:05:42.945 INFO:teuthology.orchestra.run.smithi052.stderr:==28241==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:43.010 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5da4b4e 0x5fccc1d 0x5fcce92 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:43.010 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.246 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3b8bdbb 0x3b8c15e 0xba179ce 0xb9b22dd 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.246 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.246 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xc9e2b 0x36074e8 0x3b8bf66 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.246 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3610fb6 0x3b8bee3 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.303 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x119673 0x11d39b 0x41608 0x41bbe 0x41c92 0x41f68 0x4214e 0x3ef7b 0xfd531 0xb83682d 0xba16ccc 0xba16dfe 0xba16f7a 0xba17729 0xb9b1eb4 0x9826b31 0x9a94970 0x9ae1626 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8a990950 0xffffffff8a990fd4 0xffffffff8a99b92b 0xffffffff8a99e286 0xffffffff8a99e522 0xffffffff8a99fe99 0xffffffff8a9a0afd 0xffffffff8a96ccad 0xffffffff8b2cd45f 0xffffffff8b400130 2024-09-22T04:05:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 66 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xff3cd 0x444e9 0xfb4fb 0x3767e9b 0x961a318 0x9add627 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:43.304 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.362 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x1251d7 0xfe618 0x5ca6522 0x5ca66fc 0x5ca69b3 0x5ca6978 0x5ca6aa8 0xb9ffe54 0xba0b7a4 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:43.362 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.403 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10d75f 0xdd31a 0x4ab1c 0xb8379da 0xb838488 0xba09156 0xb9ae311 0x9add337 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.403 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.403 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x133c2b 0xdd332 0x4ab1c 0xb9cc4a3 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.403 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.461 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3642781 0x9bff309 0x9bff4e1 0x9bff824 0x9adc9b5 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.461 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x3779717 0x3b8c09a 0x3b8c15e 0xb9cc607 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.461 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.461 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.502 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab92 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:43.502 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.559 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 122 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55251dc 0x9e16cf9 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:43.559 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.597 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525df6 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.597 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:43.597 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5525882 0x9e16d1b 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:43.597 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:45.068 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:45.095 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T04:05:45.121 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:45.416 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:45.416 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:45,415 seastar - Reactor backend: linux-aio 2024-09-22T04:05:45.421 INFO:teuthology.orchestra.run.smithi052.stderr:==28290==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:45.529 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab9d 0x5fccc96 0x5fcce5e 0x5fcd08f 0x5fcd365 0x601ba63 0x601bc69 0x5f9bdb6 0x5f9bf90 0x5f9c51c 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3961ae8 0x39632c2 0x3803c18 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:45.529 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:45.562 INFO:teuthology.orchestra.run.smithi052.stderr:DEBUG 2024-09-22 04:05:45,561 [shard 0:main] seastar - reactor::drain 2024-09-22T04:05:46.087 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:46.088 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:46.088 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:46.088 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:46.088 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:46.145 INFO:teuthology.orchestra.run.smithi052.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T04:05:46.149 INFO:teuthology.orchestra.run.smithi052.stderr:INFO 2024-09-22 04:05:46,149 seastar - Reactor backend: linux-aio 2024-09-22T04:05:46.159 INFO:teuthology.orchestra.run.smithi052.stderr:==28288==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T04:05:46.224 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x5fd556c 0x601b881 0x601ba89 0x601bc69 0x5f9bdb6 0x5f9c2ea 0x5de40ab 0x5dfc2ed 0x713a942 0x714c115 0x714d5b2 0x714d637 0x714d739 0x714d831 0x714da3e 0x714dc24 0x714df73 0x714e317 0x714e547 0x714e7b1 0x714ea7e 0x714ec9c 0x714ed54 0x714ee1a 0x714eefb 0x5c46eec 0x5c46ff6 0x5c470b2 0x5c4716e 0x5c4722a 0x5c47377 0x5c47502 0x5c478c3 0x5c616c6 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:46.224 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.399 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xb9bcca9 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x968cf8c 0x968d328 0x968d4ad 0x968d54f 0x968d668 0xbe289a9 0xbe28bb0 0xbe28c86 0xbe28dc1 0xbe291b5 0xbe28578 0xbe28658 0xbe2876c 0xbe2842f 0x94eb330 0x95d0372 0x9307833 0x9307a0d 0x9307b48 0x9307c02 0x9307cdc 0x9308228 0x9308500 0x93085b5 0x957fcfc 0x957ff61 0x95ae198 0x96b1946 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:46.399 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xba04f3d 0xba05178 0xba0532e 0xba0547b 0xb9adb1e 0xb9b21d1 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab2c 0x3ac7302 0x3b8c153 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cfcb3 0xba14b9d 0xb9bc06f 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.459 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4bfc6 0xb9e7536 0xb9e8829 0xba170df 0xba17729 0xb9b1eb4 0x9826b31 0x9a953a3 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 90 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156e8 0xdd312 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9cc43a 0xb9cd8da 0xb9cdb54 0xb9cdcfc 0xb9cfab7 0xb9cff02 0xb9bbe2c 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9a95997 0x9ae1492 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x931138d 0x931232c 0x93123af 0x9312434 0x9312536 0x931262e 0x93126b4 0x931276d 0x9312923 0x9312a5b 0x9312ad6 0x9312b5b 0x9312cb6 0x9312dd0 0x9312e55 0x9312f33 0x9535c26 0x95fe993 0x95feb10 0x96bf871 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10a2c4 0xfd59e 0xb83682d 0xb8368e0 0xb8369f6 0xb836ad6 0xb836d56 0xb836e0f 0xb837301 0xb83770c 0xb8379cb 0xba09729 0xba0b737 0xba0b9bf 0xba0ba6f 0xb9bb43d 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:46.517 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.575 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x422b1 0x3ecea 0xfc9b1 0x3869141 0x9826a54 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:46.575 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x156e3 0xdd312 0x4b14c 0x95ace73 0x95acffd 0x95c6829 0x9826a95 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10bf37 0x10a29f 0xfca1b 0x3869141 0xb9b16d4 0x9826b31 0x9adcd14 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.618 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x10797f 0xdd2e1 0x4ab1c 0x3ac7302 0x3b8c153 0xb9e8224 0xba170df 0xba17729 0xb9b1eb4 0x9826b31 0x9add2fa 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.675 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 123 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd2f7 0x4b14c 0x3b8bfdd 0x3b8c15e 0xb9d336b 0xb9d3620 0xb9bb9f1 0xb9bfa98 0xb9aaa36 0xb9aae1a 0x9add3c3 0x9af4f12 0x9985508 0x9925cf1 0x9257a3d 0x9258152 0x925866e 0x92587f7 0x9258a3e 0x9258d16 0x9258dcb 0x9576088 0x95762ed 0x95ae2d3 0x96b28be 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.676 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.716 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x4ab49 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc3f39d 0xb90dc6d 0xb90e4a5 0x3804428 0x2958f 0x2963f 0x35eb9a4 2024-09-22T04:05:46.716 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.813 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0xdd323 0x4ab1c 0x5525eb7 0x9e16ce0 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.813 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:46.813 INFO:teuthology.orchestra.run.smithi052.stderr:Reactor stalled for 65 ms on shard 2, in scheduling group main. Backtrace: 0x82dff 0xbc8b6ef 0xbb64079 0xbb655e4 0xbb657fa 0xbb65950 0xbb65e19 0x3e6ef 0x55253e4 0x9e16d0e 0x9e861d9 0x9eb69d1 0x9e1df64 0x993c4f7 0x985b902 0x987ffc2 0x9880121 0x98919be 0x9891b8e 0x9891e9c 0x98921ab 0x989230a 0x9892385 0x98925cc 0x9892890 0x9892920 0x98929c4 0x95a6852 0x95a6ab7 0x95af05c 0x96a1706 0xbb78627 0xbb904a5 0xbc3dd4f 0xbc5034b 0xbc509cd 0x7b13bb5 0xc5e60e4 0x5e669 0x89c51 0x10ec7f 2024-09-22T04:05:46.813 INFO:teuthology.orchestra.run.smithi052.stderr:kernel callstack: 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr:Suppressions used: 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr: count bytes template 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr: 1 8 MallocExtension::Register 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr:----------------------------------------------------- 2024-09-22T04:05:48.302 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:05:48.330 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T04:05:48.355 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T04:05:48.355 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:48.355 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T04:05:48.421 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:48.421 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T04:05:48.487 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:48.487 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T04:05:48.552 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:48.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T04:05:48.617 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:48.617 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T04:05:48.673 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T04:05:48.673 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCRl+9mUz1zKBAAQQFLQewylij6mHCekE+tPg== 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCTl+9mbawHMxAAo1eQaa7cDoA4bXSqw9Dc0A== 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCXl+9m2ez+BRAA9CnIWC2bpjI1Af5Gd2NERQ== 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCal+9mgoqzEhAA5Zsy/M6zHi0CF3uMrb6IVg== 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2024-09-22T04:05:48.737 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCRl+9mhBGnKxAAiyCp6LHDbe9sa1nnjxukmQ== 2024-09-22T04:05:48.739 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T04:05:48.840 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T04:05:48.902 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T04:05:48.964 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T04:05:49.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T04:05:49.092 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T04:05:49.092 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T04:05:49.118 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T04:05:49.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.236+0000 7f9cdde2cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.238 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.236+0000 7f9cdde2cc00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.251 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T04:05:49.276 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T04:05:49.332 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T04:05:49.332 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T04:05:49.332 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T04:05:49.374 INFO:tasks.ceph.mon.a:Started 2024-09-22T04:05:49.375 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T04:05:49.375 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T04:05:49.375 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T04:05:49.377 INFO:tasks.ceph.mgr.x:Started 2024-09-22T04:05:49.377 DEBUG:tasks.ceph:set 0 configs 2024-09-22T04:05:49.377 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T04:05:49.377 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T04:05:49.460 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-09-22T04:05:49.459+0000 7f7b72ed4c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.461 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-09-22T04:05:49.459+0000 7f7b72ed4c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.461 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-09-22T04:05:49.459+0000 7f7b72ed4c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.463 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-09-22T04:05:49.462+0000 7fb021205000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.493 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.486+0000 7f4495d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.495 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.494+0000 7f4495d51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.566 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2024-09-22T04:05:49.577 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T04:05:49.577 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T04:05:49.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T04:05:49.668 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-09-22T04:05:49.666+0000 7fb021205000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.668 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-09-22T04:05:49.666+0000 7fb021205000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.687 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.681+0000 7f3f4805c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.689 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:49.688+0000 7f3f4805c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:49.772 INFO:tasks.ceph.mon.a.smithi052.stderr:2024-09-22T04:05:49.771+0000 7f7b69bf4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T04:05:49.773 INFO:tasks.ceph.mon.a.smithi052.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T04:05:49.773 INFO:tasks.ceph.mon.a.smithi052.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T04:05:49.773 INFO:tasks.ceph.mon.a.smithi052.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T04:05:49.773 INFO:tasks.ceph.mon.a.smithi052.stderr:feature with important data. 2024-09-22T04:05:49.773 INFO:tasks.ceph.mon.a.smithi052.stderr: 2024-09-22T04:05:50.578 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T04:05:50.578 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:50.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T04:05:50.604 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:50.604 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T04:05:50.670 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:05:50.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T04:05:50.736 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 1ecf4ec1-28d7-4dac-8625-5beb076d3e59 0 2024-09-22T04:05:50.888 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:50.881+0000 7fc725711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:50.890 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:50.889+0000 7fc725711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:50.971 INFO:teuthology.orchestra.run.smithi052.stdout:0 2024-09-22T04:05:50.983 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 27a17f09-d7a4-4397-b48a-96cb280ceb3b 1 2024-09-22T04:05:51.095 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.088+0000 7ff8f0088640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.097 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.096+0000 7ff8f0088640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.175 INFO:teuthology.orchestra.run.smithi052.stdout:1 2024-09-22T04:05:51.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd new 2473b43a-fe74-4629-a38c-b12fc76c9776 2 2024-09-22T04:05:51.310 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.302+0000 7ff1988d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.312 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.310+0000 7ff1988d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.392 INFO:teuthology.orchestra.run.smithi052.stdout:2 2024-09-22T04:05:51.404 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T04:05:51.435 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T04:05:51.435 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T04:05:51.478 INFO:tasks.ceph.osd.0:Started 2024-09-22T04:05:51.478 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T04:05:51.508 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T04:05:51.508 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T04:05:51.551 INFO:tasks.ceph.osd.1:Started 2024-09-22T04:05:51.551 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T04:05:51.583 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T04:05:51.584 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T04:05:51.627 INFO:tasks.ceph.osd.2:Started 2024-09-22T04:05:51.627 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T04:05:51.801 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.795+0000 7f78e18bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.804 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:51.803+0000 7f78e18bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:51.874 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:05:51.875 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":6,"fsid":"c12a7beb-40fd-4d8a-9fc4-1739c65f3b64","created":"2024-09-22T04:05:49.480950+0000","modified":"2024-09-22T04:05:51.391843+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T04:05:51.391843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1ecf4ec1-28d7-4dac-8625-5beb076d3e59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27a17f09-d7a4-4397-b48a-96cb280ceb3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2473b43a-fe74-4629-a38c-b12fc76c9776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T04:05:51.886 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T04:05:51.886 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T04:05:52.188 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T04:05:52.294 INFO:teuthology.misc.health.smithi052.stderr:2024-09-22T04:05:52.289+0000 7f923ff91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:52.297 INFO:teuthology.misc.health.smithi052.stderr:2024-09-22T04:05:52.295+0000 7f923ff91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:52.365 INFO:teuthology.misc.health.smithi052.stdout: 2024-09-22T04:05:52.365 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":6,"fsid":"c12a7beb-40fd-4d8a-9fc4-1739c65f3b64","created":"2024-09-22T04:05:49.480950+0000","modified":"2024-09-22T04:05:51.391843+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T04:05:51.391843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1ecf4ec1-28d7-4dac-8625-5beb076d3e59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"27a17f09-d7a4-4397-b48a-96cb280ceb3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2473b43a-fe74-4629-a38c-b12fc76c9776","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T04:05:52.376 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T04:05:55.336 INFO:tasks.ceph.mgr.x.smithi052.stderr:2024-09-22T04:05:55.335+0000 7fafbe48e640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T04:05:58.678 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T04:05:58.792 INFO:teuthology.misc.health.smithi052.stderr:2024-09-22T04:05:58.787+0000 7f8f898b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:58.794 INFO:teuthology.misc.health.smithi052.stderr:2024-09-22T04:05:58.794+0000 7f8f898b4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:59.212 INFO:teuthology.misc.health.smithi052.stdout: 2024-09-22T04:05:59.212 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"c12a7beb-40fd-4d8a-9fc4-1739c65f3b64","created":"2024-09-22T04:05:49.480950+0000","modified":"2024-09-22T04:05:58.393509+0000","last_up_change":"2024-09-22T04:05:58.393509+0000","last_in_change":"2024-09-22T04:05:51.391843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1ecf4ec1-28d7-4dac-8625-5beb076d3e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6813","nonce":2444183389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2444183389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2444183389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2444183389}]},"public_addr":"172.21.15.52:6813/2444183389","cluster_addr":"172.21.15.52:6814/2444183389","heartbeat_back_addr":"172.21.15.52:6812/2444183389","heartbeat_front_addr":"172.21.15.52:6810/2444183389","state":["exists","up"]},{"osd":1,"uuid":"27a17f09-d7a4-4397-b48a-96cb280ceb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6803","nonce":3833099718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":3833099718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":3833099718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":3833099718}]},"public_addr":"172.21.15.52:6803/3833099718","cluster_addr":"172.21.15.52:6802/3833099718","heartbeat_back_addr":"172.21.15.52:6806/3833099718","heartbeat_front_addr":"172.21.15.52:6805/3833099718","state":["exists","up"]},{"osd":2,"uuid":"2473b43a-fe74-4629-a38c-b12fc76c9776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6807","nonce":2955551813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2955551813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6809","nonce":2955551813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":2955551813}]},"public_addr":"172.21.15.52:6807/2955551813","cluster_addr":"172.21.15.52:6808/2955551813","heartbeat_back_addr":"172.21.15.52:6809/2955551813","heartbeat_front_addr":"172.21.15.52:6811/2955551813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T04:05:59.230 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T04:05:59.231 INFO:tasks.ceph:Creating RBD pool 2024-09-22T04:05:59.231 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T04:05:59.354 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:59.345+0000 7f9d77d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:05:59.357 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:05:59.357+0000 7f9d77d91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:00.404 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'rbd' created 2024-09-22T04:06:00.430 DEBUG:teuthology.orchestra.run.smithi052:> rbd --cluster ceph pool init rbd 2024-09-22T04:06:00.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:00.543+0000 7f4804610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:00.544 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:00.544+0000 7f4804610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:00.548 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:00.548+0000 7f4804610ac0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:03.471 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T04:06:03.472 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-22T04:06:03.472 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T04:06:03.593 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:03.588+0000 7fa960751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:03.595 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:03.594+0000 7fa960751640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:03.794 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:03.806 INFO:teuthology.orchestra.run.smithi052.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2024-09-22T04:06:03.806 INFO:tasks.ceph_manager:config epoch is 1 2024-09-22T04:06:03.806 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T04:06:03.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T04:06:03.807 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T04:06:03.919 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:03.915+0000 7faf9564f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:03.921 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:03.921+0000 7faf9564f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.177 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:04.189 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":3462761664},{"type":"v1","addr":"172.21.15.52:6801","nonce":3462761664}]},"active_addr":"172.21.15.52:6801/3462761664","active_change":"2024-09-22T04:05:54.322534+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":4067757529}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1160095402}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2085472398}]}]} 2024-09-22T04:06:04.192 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T04:06:04.193 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T04:06:04.193 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T04:06:04.306 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.302+0000 7fb301f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.308 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.308+0000 7fb301f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.506 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:04.507 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":14,"fsid":"c12a7beb-40fd-4d8a-9fc4-1739c65f3b64","created":"2024-09-22T04:05:49.480950+0000","modified":"2024-09-22T04:06:03.409855+0000","last_up_change":"2024-09-22T04:05:58.393509+0000","last_in_change":"2024-09-22T04:05:51.391843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T04:05:59.566390+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T04:06:00.347591+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ecf4ec1-28d7-4dac-8625-5beb076d3e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6813","nonce":2444183389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2444183389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2444183389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2444183389}]},"public_addr":"172.21.15.52:6813/2444183389","cluster_addr":"172.21.15.52:6814/2444183389","heartbeat_back_addr":"172.21.15.52:6812/2444183389","heartbeat_front_addr":"172.21.15.52:6810/2444183389","state":["exists","up"]},{"osd":1,"uuid":"27a17f09-d7a4-4397-b48a-96cb280ceb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6803","nonce":3833099718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":3833099718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":3833099718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":3833099718}]},"public_addr":"172.21.15.52:6803/3833099718","cluster_addr":"172.21.15.52:6802/3833099718","heartbeat_back_addr":"172.21.15.52:6806/3833099718","heartbeat_front_addr":"172.21.15.52:6805/3833099718","state":["exists","up"]},{"osd":2,"uuid":"2473b43a-fe74-4629-a38c-b12fc76c9776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6807","nonce":2955551813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2955551813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6809","nonce":2955551813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":2955551813}]},"public_addr":"172.21.15.52:6807/2955551813","cluster_addr":"172.21.15.52:6808/2955551813","heartbeat_back_addr":"172.21.15.52:6809/2955551813","heartbeat_front_addr":"172.21.15.52:6811/2955551813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T04:06:04.520 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T04:06:04.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T04:06:04.636 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.632+0000 7f0db32bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.638+0000 7f0db32bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.833 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:04.833 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":14,"fsid":"c12a7beb-40fd-4d8a-9fc4-1739c65f3b64","created":"2024-09-22T04:05:49.480950+0000","modified":"2024-09-22T04:06:03.409855+0000","last_up_change":"2024-09-22T04:05:58.393509+0000","last_in_change":"2024-09-22T04:05:51.391843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-22T04:05:59.566390+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-09-22T04:06:00.347591+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1ecf4ec1-28d7-4dac-8625-5beb076d3e59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6813","nonce":2444183389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":2444183389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":2444183389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":2444183389}]},"public_addr":"172.21.15.52:6813/2444183389","cluster_addr":"172.21.15.52:6814/2444183389","heartbeat_back_addr":"172.21.15.52:6812/2444183389","heartbeat_front_addr":"172.21.15.52:6810/2444183389","state":["exists","up"]},{"osd":1,"uuid":"27a17f09-d7a4-4397-b48a-96cb280ceb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6803","nonce":3833099718}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":3833099718}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":3833099718}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6805","nonce":3833099718}]},"public_addr":"172.21.15.52:6803/3833099718","cluster_addr":"172.21.15.52:6802/3833099718","heartbeat_back_addr":"172.21.15.52:6806/3833099718","heartbeat_front_addr":"172.21.15.52:6805/3833099718","state":["exists","up"]},{"osd":2,"uuid":"2473b43a-fe74-4629-a38c-b12fc76c9776","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6807","nonce":2955551813}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":2955551813}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6809","nonce":2955551813}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6811","nonce":2955551813}]},"public_addr":"172.21.15.52:6807/2955551813","cluster_addr":"172.21.15.52:6808/2955551813","heartbeat_back_addr":"172.21.15.52:6809/2955551813","heartbeat_front_addr":"172.21.15.52:6811/2955551813","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T04:06:04.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T04:06:04.847 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T04:06:04.848 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T04:06:04.966 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.961+0000 7f2160511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.962+0000 7f5d7f711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.967 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.962+0000 7fbb184fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.969 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.969+0000 7f2160511640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.969+0000 7fbb184fd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:04.970 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:04.969+0000 7f5d7f711640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.000 INFO:teuthology.orchestra.run.smithi052.stdout:34359738376 2024-09-22T04:06:05.001 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T04:06:05.002 INFO:teuthology.orchestra.run.smithi052.stdout:42949672966 2024-09-22T04:06:05.002 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T04:06:05.003 INFO:teuthology.orchestra.run.smithi052.stdout:42949672966 2024-09-22T04:06:05.003 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T04:06:05.120 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.115+0000 7ff5cdabd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.116+0000 7fb6f669c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.121 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.116+0000 7f6397b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.123+0000 7ff5cdabd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.123 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.123+0000 7fb6f669c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.124 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:05.124+0000 7f6397b11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:05.338 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:05.343 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-22T04:06:05.349 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:05.352 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T04:06:05.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T04:06:05.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T04:06:06.353 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T04:06:06.359 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T04:06:06.365 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T04:06:06.491 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.485+0000 7f3259191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.494 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.493+0000 7f3259191640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.521+0000 7fdc83d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.526 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.521+0000 7fa635693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.528+0000 7fdc83d11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.528 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:06.528+0000 7fa635693640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:06.709 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:06.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T04:06:06.733 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:06.745 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-22T04:06:06.747 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T04:06:06.758 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T04:06:07.723 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T04:06:07.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T04:06:07.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T04:06:07.860 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.853+0000 7efdb6791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:07.863 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.863+0000 7efdb6791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:07.895 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.890+0000 7f47c269c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:07.898 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.897+0000 7f47c269c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:07.907 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.898+0000 7f4c29591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:07.910 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:07.910+0000 7f4c29591640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:08.099 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2024-09-22T04:06:08.110 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:08.112 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738372 for osd.1 2024-09-22T04:06:08.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T04:06:08.145 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:08.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T04:06:09.112 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T04:06:09.125 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T04:06:09.160 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T04:06:09.247 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.240+0000 7fd4eaad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.249 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.249+0000 7fd4eaad1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.284 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.278+0000 7f6168991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.287 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.286+0000 7f6168991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.323 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.319+0000 7f2177791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.326 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:09.325+0000 7f2177791640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:09.451 INFO:teuthology.orchestra.run.smithi052.stdout:34359738377 2024-09-22T04:06:09.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738376 got 34359738377 for osd.1 2024-09-22T04:06:09.465 DEBUG:teuthology.parallel:result is None 2024-09-22T04:06:09.490 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:09.503 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.2 2024-09-22T04:06:09.532 INFO:teuthology.orchestra.run.smithi052.stdout:42949672964 2024-09-22T04:06:09.546 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672964 for osd.0 2024-09-22T04:06:10.504 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T04:06:10.546 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T04:06:10.638 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:10.631+0000 7fa002e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:10.640 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:10.640+0000 7fa002e6c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:10.672 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:10.667+0000 7f64c7efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:10.674 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:10.674+0000 7f64c7efd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:10.847 INFO:teuthology.orchestra.run.smithi052.stdout:42949672969 2024-09-22T04:06:10.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.2 2024-09-22T04:06:10.861 DEBUG:teuthology.parallel:result is None 2024-09-22T04:06:10.880 INFO:teuthology.orchestra.run.smithi052.stdout:42949672969 2024-09-22T04:06:10.894 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672966 got 42949672969 for osd.0 2024-09-22T04:06:10.894 DEBUG:teuthology.parallel:result is None 2024-09-22T04:06:10.894 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T04:06:10.894 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T04:06:11.047 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.044+0000 7f6618391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.050 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.049+0000 7f6618391640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.244 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:11.244 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-22T04:06:11.256 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T04:06:10.328749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.797917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434665+0000","last_change":"2024-09-22T04:06:03.441458+0000","last_active":"2024-09-22T04:06:03.434665+0000","last_peered":"2024-09-22T04:06:03.434665+0000","last_clean":"2024-09-22T04:06:03.434665+0000","last_became_active":"2024-09-22T04:06:01.575292+0000","last_became_peered":"2024-09-22T04:06:01.575292+0000","last_unstale":"2024-09-22T04:06:03.434665+0000","last_undegraded":"2024-09-22T04:06:03.434665+0000","last_fullsized":"2024-09-22T04:06:03.434665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.431387+0000","last_change":"2024-09-22T04:06:03.436607+0000","last_active":"2024-09-22T04:06:03.431387+0000","last_peered":"2024-09-22T04:06:03.431387+0000","last_clean":"2024-09-22T04:06:03.431387+0000","last_became_active":"2024-09-22T04:06:01.626482+0000","last_became_peered":"2024-09-22T04:06:01.626482+0000","last_unstale":"2024-09-22T04:06:03.431387+0000","last_undegraded":"2024-09-22T04:06:03.431387+0000","last_fullsized":"2024-09-22T04:06:03.431387+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.469183+0000","last_change":"2024-09-22T04:06:03.476984+0000","last_active":"2024-09-22T04:06:03.469183+0000","last_peered":"2024-09-22T04:06:03.469183+0000","last_clean":"2024-09-22T04:06:03.469183+0000","last_became_active":"2024-09-22T04:06:01.589000+0000","last_became_peered":"2024-09-22T04:06:01.589000+0000","last_unstale":"2024-09-22T04:06:03.469183+0000","last_undegraded":"2024-09-22T04:06:03.469183+0000","last_fullsized":"2024-09-22T04:06:03.469183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434196+0000","last_change":"2024-09-22T04:06:03.436757+0000","last_active":"2024-09-22T04:06:03.434196+0000","last_peered":"2024-09-22T04:06:03.434196+0000","last_clean":"2024-09-22T04:06:03.434196+0000","last_became_active":"2024-09-22T04:06:01.592995+0000","last_became_peered":"2024-09-22T04:06:01.592995+0000","last_unstale":"2024-09-22T04:06:03.434196+0000","last_undegraded":"2024-09-22T04:06:03.434196+0000","last_fullsized":"2024-09-22T04:06:03.434196+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.473918+0000","last_change":"2024-09-22T04:06:01.594086+0000","last_active":"2024-09-22T04:06:03.473918+0000","last_peered":"2024-09-22T04:06:03.473918+0000","last_clean":"2024-09-22T04:06:03.473918+0000","last_became_active":"2024-09-22T04:06:01.550628+0000","last_became_peered":"2024-09-22T04:06:01.550628+0000","last_unstale":"2024-09-22T04:06:03.473918+0000","last_undegraded":"2024-09-22T04:06:03.473918+0000","last_fullsized":"2024-09-22T04:06:03.473918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434237+0000","last_change":"2024-09-22T04:06:03.441571+0000","last_active":"2024-09-22T04:06:03.434237+0000","last_peered":"2024-09-22T04:06:03.434237+0000","last_clean":"2024-09-22T04:06:03.434237+0000","last_became_active":"2024-09-22T04:06:01.666977+0000","last_became_peered":"2024-09-22T04:06:01.666977+0000","last_unstale":"2024-09-22T04:06:03.434237+0000","last_undegraded":"2024-09-22T04:06:03.434237+0000","last_fullsized":"2024-09-22T04:06:03.434237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.442209+0000","last_change":"2024-09-22T04:06:03.442209+0000","last_active":"2024-09-22T04:06:03.442209+0000","last_peered":"2024-09-22T04:06:03.442209+0000","last_clean":"2024-09-22T04:06:03.442209+0000","last_became_active":"2024-09-22T04:06:01.516641+0000","last_became_peered":"2024-09-22T04:06:01.516641+0000","last_unstale":"2024-09-22T04:06:03.442209+0000","last_undegraded":"2024-09-22T04:06:03.442209+0000","last_fullsized":"2024-09-22T04:06:03.442209+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.470522+0000","last_change":"2024-09-22T04:06:03.477604+0000","last_active":"2024-09-22T04:06:03.470522+0000","last_peered":"2024-09-22T04:06:03.470522+0000","last_clean":"2024-09-22T04:06:03.470522+0000","last_became_active":"2024-09-22T04:06:01.702110+0000","last_became_peered":"2024-09-22T04:06:01.702110+0000","last_unstale":"2024-09-22T04:06:03.470522+0000","last_undegraded":"2024-09-22T04:06:03.470522+0000","last_fullsized":"2024-09-22T04:06:03.470522+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.431099+0000","last_change":"2024-09-22T04:06:03.436391+0000","last_active":"2024-09-22T04:06:03.431099+0000","last_peered":"2024-09-22T04:06:03.431099+0000","last_clean":"2024-09-22T04:06:03.431099+0000","last_became_active":"2024-09-22T04:06:01.614541+0000","last_became_peered":"2024-09-22T04:06:01.614541+0000","last_unstale":"2024-09-22T04:06:03.431099+0000","last_undegraded":"2024-09-22T04:06:03.431099+0000","last_fullsized":"2024-09-22T04:06:03.431099+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T04:06:11.257 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T04:06:11.370 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.367+0000 7f0fa8f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.372 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.372+0000 7f0fa8f91640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.566 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:11.567 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2024-09-22T04:06:11.579 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-22T04:06:10.328749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3432,"kb_used_data":3432,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889752,"statfs":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.797917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434665+0000","last_change":"2024-09-22T04:06:03.441458+0000","last_active":"2024-09-22T04:06:03.434665+0000","last_peered":"2024-09-22T04:06:03.434665+0000","last_clean":"2024-09-22T04:06:03.434665+0000","last_became_active":"2024-09-22T04:06:01.575292+0000","last_became_peered":"2024-09-22T04:06:01.575292+0000","last_unstale":"2024-09-22T04:06:03.434665+0000","last_undegraded":"2024-09-22T04:06:03.434665+0000","last_fullsized":"2024-09-22T04:06:03.434665+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.431387+0000","last_change":"2024-09-22T04:06:03.436607+0000","last_active":"2024-09-22T04:06:03.431387+0000","last_peered":"2024-09-22T04:06:03.431387+0000","last_clean":"2024-09-22T04:06:03.431387+0000","last_became_active":"2024-09-22T04:06:01.626482+0000","last_became_peered":"2024-09-22T04:06:01.626482+0000","last_unstale":"2024-09-22T04:06:03.431387+0000","last_undegraded":"2024-09-22T04:06:03.431387+0000","last_fullsized":"2024-09-22T04:06:03.431387+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.469183+0000","last_change":"2024-09-22T04:06:03.476984+0000","last_active":"2024-09-22T04:06:03.469183+0000","last_peered":"2024-09-22T04:06:03.469183+0000","last_clean":"2024-09-22T04:06:03.469183+0000","last_became_active":"2024-09-22T04:06:01.589000+0000","last_became_peered":"2024-09-22T04:06:01.589000+0000","last_unstale":"2024-09-22T04:06:03.469183+0000","last_undegraded":"2024-09-22T04:06:03.469183+0000","last_fullsized":"2024-09-22T04:06:03.469183+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434196+0000","last_change":"2024-09-22T04:06:03.436757+0000","last_active":"2024-09-22T04:06:03.434196+0000","last_peered":"2024-09-22T04:06:03.434196+0000","last_clean":"2024-09-22T04:06:03.434196+0000","last_became_active":"2024-09-22T04:06:01.592995+0000","last_became_peered":"2024-09-22T04:06:01.592995+0000","last_unstale":"2024-09-22T04:06:03.434196+0000","last_undegraded":"2024-09-22T04:06:03.434196+0000","last_fullsized":"2024-09-22T04:06:03.434196+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"14'52","reported_seq":81,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.473918+0000","last_change":"2024-09-22T04:06:01.594086+0000","last_active":"2024-09-22T04:06:03.473918+0000","last_peered":"2024-09-22T04:06:03.473918+0000","last_clean":"2024-09-22T04:06:03.473918+0000","last_became_active":"2024-09-22T04:06:01.550628+0000","last_became_peered":"2024-09-22T04:06:01.550628+0000","last_unstale":"2024-09-22T04:06:03.473918+0000","last_undegraded":"2024-09-22T04:06:03.473918+0000","last_fullsized":"2024-09-22T04:06:03.473918+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.434237+0000","last_change":"2024-09-22T04:06:03.441571+0000","last_active":"2024-09-22T04:06:03.434237+0000","last_peered":"2024-09-22T04:06:03.434237+0000","last_clean":"2024-09-22T04:06:03.434237+0000","last_became_active":"2024-09-22T04:06:01.666977+0000","last_became_peered":"2024-09-22T04:06:01.666977+0000","last_unstale":"2024-09-22T04:06:03.434237+0000","last_undegraded":"2024-09-22T04:06:03.434237+0000","last_fullsized":"2024-09-22T04:06:03.434237+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":17,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.442209+0000","last_change":"2024-09-22T04:06:03.442209+0000","last_active":"2024-09-22T04:06:03.442209+0000","last_peered":"2024-09-22T04:06:03.442209+0000","last_clean":"2024-09-22T04:06:03.442209+0000","last_became_active":"2024-09-22T04:06:01.516641+0000","last_became_peered":"2024-09-22T04:06:01.516641+0000","last_unstale":"2024-09-22T04:06:03.442209+0000","last_undegraded":"2024-09-22T04:06:03.442209+0000","last_fullsized":"2024-09-22T04:06:03.442209+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.470522+0000","last_change":"2024-09-22T04:06:03.477604+0000","last_active":"2024-09-22T04:06:03.470522+0000","last_peered":"2024-09-22T04:06:03.470522+0000","last_clean":"2024-09-22T04:06:03.470522+0000","last_became_active":"2024-09-22T04:06:01.702110+0000","last_became_peered":"2024-09-22T04:06:01.702110+0000","last_unstale":"2024-09-22T04:06:03.470522+0000","last_undegraded":"2024-09-22T04:06:03.470522+0000","last_fullsized":"2024-09-22T04:06:03.470522+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-22T04:06:03.431099+0000","last_change":"2024-09-22T04:06:03.436391+0000","last_active":"2024-09-22T04:06:03.431099+0000","last_peered":"2024-09-22T04:06:03.431099+0000","last_clean":"2024-09-22T04:06:03.431099+0000","last_became_active":"2024-09-22T04:06:01.614541+0000","last_became_peered":"2024-09-22T04:06:01.614541+0000","last_unstale":"2024-09-22T04:06:03.431099+0000","last_undegraded":"2024-09-22T04:06:03.431099+0000","last_fullsized":"2024-09-22T04:06:03.431099+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T04:06:00.399209+0000","last_clean_scrub_stamp":"2024-09-22T04:06:00.399209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655106048,"internally_reserved":0,"allocated":3514368,"data_stored":3514368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1292,"kb_used_data":1292,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296436,"statfs":{"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":652,"kb_used_data":652,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297076,"statfs":{"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1488,"kb_used_data":1488,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296240,"statfs":{"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":53552873472,"available":53552205824,"internally_reserved":0,"allocated":667648,"data_stored":667648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":53552873472,"available":53551349760,"internally_reserved":0,"allocated":1523712,"data_stored":1523712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53551550464,"internally_reserved":0,"allocated":1323008,"data_stored":1323008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T04:06:11.580 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T04:06:11.580 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T04:06:11.580 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T04:06:11.581 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T04:06:11.692 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.689+0000 7f4fac688640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.694 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:06:11.694+0000 7f4fac688640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:06:11.930 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:11.930 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T04:06:11.943 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T04:06:11.943 INFO:teuthology.run_tasks:Running task cbt... 2024-09-22T04:06:11.957 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi052.front.sepia.ceph.com', 'osds': ['ubuntu@smithi052.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi052.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi052.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi052.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-22T04:06:11.957 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-22T04:06:11.975 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:06:11.975 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T04:06:12.030 DEBUG:teuthology.orchestra.run.smithi052:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-22T04:06:12.086 INFO:teuthology.orchestra.run.smithi052.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-22T04:06:12.397 INFO:tasks.cbt:Installing collectl 2024-09-22T04:06:12.397 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-22T04:06:12.493 INFO:teuthology.orchestra.run.smithi052.stderr:--2024-09-22 04:06:12-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-22T04:06:12.660 INFO:teuthology.orchestra.run.smithi052.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2024-09-22T04:06:12.673 INFO:teuthology.orchestra.run.smithi052.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-09-22T04:06:12.908 INFO:teuthology.orchestra.run.smithi052.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T04:06:12.908 INFO:teuthology.orchestra.run.smithi052.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75e0tVepi9md74cc0wPaN7I7yuyyibk2vaYbp0wT7bLhMaqAQ59W72LnbC1Bf9s8CHBHNZ4HkOleTaaGxiEIAuPurw%3D%3D&use_mirror=pilotfiber&r= [following] 2024-09-22T04:06:12.908 INFO:teuthology.orchestra.run.smithi052.stderr:--2024-09-22 04:06:12-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75e0tVepi9md74cc0wPaN7I7yuyyibk2vaYbp0wT7bLhMaqAQ59W72LnbC1Bf9s8CHBHNZ4HkOleTaaGxiEIAuPurw%3D%3D&use_mirror=pilotfiber&r= 2024-09-22T04:06:12.908 INFO:teuthology.orchestra.run.smithi052.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-22T04:06:12.985 INFO:teuthology.orchestra.run.smithi052.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-22T04:06:13.289 INFO:teuthology.orchestra.run.smithi052.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T04:06:13.289 INFO:teuthology.orchestra.run.smithi052.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-22T04:06:13.290 INFO:teuthology.orchestra.run.smithi052.stderr:--2024-09-22 04:06:13-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-22T04:06:13.299 INFO:teuthology.orchestra.run.smithi052.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2024-09-22T04:06:13.313 INFO:teuthology.orchestra.run.smithi052.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2024-09-22T04:06:13.415 INFO:teuthology.orchestra.run.smithi052.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-22T04:06:13.415 INFO:teuthology.orchestra.run.smithi052.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-22T04:06:13.416 INFO:teuthology.orchestra.run.smithi052.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-22T04:06:13.416 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:06:13.440 INFO:teuthology.orchestra.run.smithi052.stderr: 0K .......... .......... .......... .......... .......... 8% 1.98M 0s 2024-09-22T04:06:13.452 INFO:teuthology.orchestra.run.smithi052.stderr: 50K .......... .......... .......... .......... .......... 16% 4.04M 0s 2024-09-22T04:06:13.454 INFO:teuthology.orchestra.run.smithi052.stderr: 100K .......... .......... .......... .......... .......... 24% 23.6M 0s 2024-09-22T04:06:13.455 INFO:teuthology.orchestra.run.smithi052.stderr: 150K .......... .......... .......... .......... .......... 32% 250M 0s 2024-09-22T04:06:13.466 INFO:teuthology.orchestra.run.smithi052.stderr: 200K .......... .......... .......... .......... .......... 41% 4.38M 0s 2024-09-22T04:06:13.469 INFO:teuthology.orchestra.run.smithi052.stderr: 250K .......... .......... .......... .......... .......... 49% 25.4M 0s 2024-09-22T04:06:13.469 INFO:teuthology.orchestra.run.smithi052.stderr: 300K .......... .......... .......... .......... .......... 57% 304M 0s 2024-09-22T04:06:13.469 INFO:teuthology.orchestra.run.smithi052.stderr: 350K .......... .......... .......... .......... .......... 65% 269M 0s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 400K .......... .......... .......... .......... .......... 74% 4.52M 0s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 450K .......... .......... .......... .......... .......... 82% 292M 0s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 500K .......... .......... .......... .......... .......... 90% 313M 0s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 550K .......... .......... .......... .......... .......... 98% 296M 0s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 600K ...... 100% 11.4T=0.06s 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22 04:06:13 (9.29 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-22T04:06:13.481 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:06:13.511 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/ 2024-09-22T04:06:13.511 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/collectl.conf 2024-09-22T04:06:13.511 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/ 2024-09-22T04:06:13.511 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Socket.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/colmux.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Import.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/style.css 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Process.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-22T04:06:13.512 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Export.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Startup.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Performance.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Messages.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Logging.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-22T04:06:13.513 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Misc.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Examples.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Data.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-22T04:06:13.514 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Playback.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Hello.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Network.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Memory.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/Disks.html 2024-09-22T04:06:13.515 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/service/ 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/service/collectl.service 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/initd/ 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/initd/collectl 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/misc.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/gexpr.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/statsd.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/envrules.std 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/hello.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/vmsum.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/proctree.ph 2024-09-22T04:06:13.516 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/README-WINDOWS 2024-09-22T04:06:13.517 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/formatit.ph 2024-09-22T04:06:13.518 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/colmux 2024-09-22T04:06:13.518 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/INSTALL 2024-09-22T04:06:13.518 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/graphite.ph 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/vmstat.ph 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/README 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/man1/ 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/man1/collectl.1 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/man1/colmux.1 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/lexpr.ph 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/client.pl 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/collectl 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/COPYING 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/vnet.ph 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/GPL 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/UNINSTALL 2024-09-22T04:06:13.519 INFO:teuthology.orchestra.run.smithi052.stdout:collectl-4.3.1/ARTISTIC 2024-09-22T04:06:13.575 INFO:teuthology.orchestra.run.smithi052.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-22T04:06:13.577 DEBUG:teuthology.orchestra.run.smithi052:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-22T04:06:14.196 INFO:teuthology.orchestra.run.smithi052.stdout:Last metadata expiration check: 0:03:03 ago on Sun 22 Sep 2024 04:03:11 AM UTC. 2024-09-22T04:06:14.290 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-22T04:06:14.292 INFO:teuthology.orchestra.run.smithi052.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-22T04:06:14.343 INFO:teuthology.orchestra.run.smithi052.stdout:Dependencies resolved. 2024-09-22T04:06:14.343 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:06:14.343 INFO:teuthology.orchestra.run.smithi052.stdout: Package Arch Version Repository Size 2024-09-22T04:06:14.343 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Installing: 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: librbd-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 24 k 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Installing dependencies: 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: libradospp-devel x86_64 2:19.3.0-5092.g62da54b4.el9 ceph 31 k 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction Summary 2024-09-22T04:06:14.344 INFO:teuthology.orchestra.run.smithi052.stdout:================================================================================ 2024-09-22T04:06:14.345 INFO:teuthology.orchestra.run.smithi052.stdout:Install 5 Packages 2024-09-22T04:06:14.345 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:14.345 INFO:teuthology.orchestra.run.smithi052.stdout:Total download size: 2.4 M 2024-09-22T04:06:14.346 INFO:teuthology.orchestra.run.smithi052.stdout:Installed size: 11 M 2024-09-22T04:06:14.346 INFO:teuthology.orchestra.run.smithi052.stdout:Downloading Packages: 2024-09-22T04:06:14.920 INFO:teuthology.orchestra.run.smithi052.stdout:(1/5): libradospp-devel-19.3.0-5092.g62da54b4.e 166 kB/s | 31 kB 00:00 2024-09-22T04:06:14.948 INFO:teuthology.orchestra.run.smithi052.stdout:(2/5): librbd-devel-19.3.0-5092.g62da54b4.el9.x 111 kB/s | 24 kB 00:00 2024-09-22T04:06:15.007 INFO:teuthology.orchestra.run.smithi052.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 225 kB/s | 13 kB 00:00 2024-09-22T04:06:15.040 INFO:teuthology.orchestra.run.smithi052.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 984 kB/s | 118 kB 00:00 2024-09-22T04:06:15.383 INFO:teuthology.orchestra.run.smithi052.stdout:(5/5): perf-5.14.0-508.el9.x86_64.rpm 3.4 MB/s | 2.2 MB 00:00 2024-09-22T04:06:15.386 INFO:teuthology.orchestra.run.smithi052.stdout:-------------------------------------------------------------------------------- 2024-09-22T04:06:15.386 INFO:teuthology.orchestra.run.smithi052.stdout:Total 2.3 MB/s | 2.4 MB 00:01 2024-09-22T04:06:15.418 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction check 2024-09-22T04:06:15.442 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction check succeeded. 2024-09-22T04:06:15.442 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction test 2024-09-22T04:06:15.581 INFO:teuthology.orchestra.run.smithi052.stdout:Transaction test succeeded. 2024-09-22T04:06:15.582 INFO:teuthology.orchestra.run.smithi052.stdout:Running transaction 2024-09-22T04:06:15.940 INFO:teuthology.orchestra.run.smithi052.stdout: Preparing : 1/1 2024-09-22T04:06:16.002 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-22T04:06:16.032 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-22T04:06:16.063 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 3/5 2024-09-22T04:06:16.410 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 4/5 2024-09-22T04:06:16.488 INFO:teuthology.orchestra.run.smithi052.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T04:06:18.146 INFO:teuthology.orchestra.run.smithi052.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T04:06:18.146 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 1/5 2024-09-22T04:06:18.147 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2/5 2024-09-22T04:06:18.147 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-22T04:06:18.147 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout:Installed: 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: libradospp-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: librbd-devel-2:19.3.0-5092.g62da54b4.el9.x86_64 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: pdsh-2.34-7.el9.x86_64 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-22T04:06:18.499 INFO:teuthology.orchestra.run.smithi052.stdout: perf-5.14.0-508.el9.x86_64 2024-09-22T04:06:18.500 INFO:teuthology.orchestra.run.smithi052.stdout: 2024-09-22T04:06:18.500 INFO:teuthology.orchestra.run.smithi052.stdout:Complete! 2024-09-22T04:06:18.875 INFO:tasks.cbt:benchmark: radosbench 2024-09-22T04:06:18.876 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-22T04:06:18.902 DEBUG:teuthology.orchestra.run.smithi052:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - Settings.cluster: 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'clients': ['ubuntu@smithi052.front.sepia.ceph.com'], 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'head': 'ubuntu@smithi052.front.sepia.ceph.com', 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'is_teuthology': True, 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'iterations': 1, 2024-09-22T04:06:19.279 INFO:teuthology.orchestra.run.smithi052.stderr: 'mons': ['ubuntu@smithi052.front.sepia.ceph.com'], 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'osds': ['ubuntu@smithi052.front.sepia.ceph.com'], 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'osds_per_node': 3, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'pid_dir': '/var/run/ceph', 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'pgp_size': 256, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'replication': 'replicated'}}, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'rebuild_every_test': False, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'rgws': {'ubuntu@smithi052.front.sepia.ceph.com': None}, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'use_existing': True, 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr: 'user': 'ubuntu'} 2024-09-22T04:06:19.280 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T04:06:19.300 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-22T04:06:19.302 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2024-09-22T04:06:19.306 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T04:06:19.307 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2024-09-22T04:06:19.310 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T04:06:19.310 INFO:teuthology.orchestra.run.smithi052.stderr:04:06:19 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T04:07:19.358 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T04:07:19.417 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T04:07:19.472 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-09-22T04:07:19.474 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-09-22T04:07:19.476 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2024-09-22T04:07:19.491 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi052.front.sepia.ceph.com ; done 2024-09-22T04:07:19.496 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-09-22T04:07:19.526 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - WARNING - cbt - sudo killall -9 rados 2024-09-22T04:07:19.526 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - WARNING - cbt - error 1 seen, continuing anyway... 2024-09-22T04:07:19.526 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T04:07:19.546 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2024-09-22T04:07:19.546 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T04:07:19.546 INFO:teuthology.orchestra.run.smithi052.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T04:07:19.546 INFO:teuthology.orchestra.run.smithi052.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': True, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2024-09-22T04:07:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T04:07:19.547 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T04:07:19.560 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2024-09-22T04:07:19.563 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T04:07:19.564 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2024-09-22T04:07:19.568 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T04:07:19.568 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T04:07:19.568 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-09-22T04:07:19.893 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:19 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-09-22T04:07:20.496 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-09-22T04:07:21.499 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:21 - INFO - cbt - Checking Health after pool creation. 2024-09-22T04:07:26.505 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:26 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T04:07:27.041 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:27 - INFO - cbt - Final Pool Health Check. 2024-09-22T04:07:32.047 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T04:07:32.398 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T04:07:32.434 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T04:07:32.483 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-09-22T04:07:32.485 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-09-22T04:07:32.487 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-22T04:07:32.847 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-22T04:07:33.409 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-22T04:07:33.771 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:33 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2024-09-22T04:07:33.841 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:33 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2024-09-22T04:07:34.719 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2024-09-22T04:07:34.722 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T04:07:34.723 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2024-09-22T04:07:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T04:07:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T04:07:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - INFO - cbt - Running radosbench write test. 2024-09-22T04:07:34.727 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2024-09-22T04:07:34.728 INFO:teuthology.orchestra.run.smithi052.stderr:04:07:34 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2024-09-22T04:08:35.921 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:35 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T04:08:36.071 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T04:08:36.119 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2024-09-22T04:08:36.121 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2024-09-22T04:08:36.123 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2024-09-22T04:08:36.497 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2024-09-22T04:08:36.511 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi052.front.sepia.ceph.com ; done 2024-09-22T04:08:36.655 INFO:teuthology.orchestra.run.smithi052.stderr:04:08:36 - INFO - cbt - Convert results to json format. 2024-09-22T04:08:36.678 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-09-22T04:08:36.687 DEBUG:teuthology.orchestra.run.smithi052:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-09-22T04:08:36.873 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-09-22T04:08:36.873 INFO:tasks.cbt_performance:reading cbt preferences 2024-09-22T04:08:36.873 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-09-22T04:08:36.945 INFO:teuthology.orchestra.run.smithi052.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi052.front.sepia.ceph.com 2024-09-22T04:08:36.945 INFO:teuthology.orchestra.run.smithi052.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi052.front.sepia.ceph.com 2024-09-22T04:08:36.946 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:08:36.946 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi052.front.sepia.ceph.com of=/dev/stdout 2024-09-22T04:08:36.963 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T04:08:36.963 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-09-22T04:08:37.020 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T04:08:37.020 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T04:08:37.020 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2024-09-22T04:08:37.020 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi052.front.sepia.ceph.com of=/dev/stdout 2024-09-22T04:08:37.076 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T04:08:37.077 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-09-22T04:08:37.132 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T04:08:37.132 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T04:08:37.140 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T04:08:37.146 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T04:08:37.146 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T04:08:37.154 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T04:08:37.307 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:08:37.299+0000 7f8025991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:08:37.309 INFO:teuthology.orchestra.run.smithi052.stderr:2024-09-22T04:08:37.308+0000 7f8025991640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T04:08:37.536 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T04:08:37.536 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T04:08:37.537 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T04:08:37.537 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T04:08:37.629 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T04:08:37.629 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T04:08:37.629 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T04:08:37.719 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T04:08:37.719 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T04:08:37.719 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T04:08:37.813 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T04:08:37.813 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T04:08:37.813 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T04:08:37.814 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T04:08:37.843 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T04:08:37.843 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T04:08:37.843 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T04:08:37.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T04:08:37.894 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T04:08:37.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T04:08:37.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T04:08:37.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-22T04:08:37.895 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T04:08:37.895 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T04:08:37.895 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-22T04:08:37.972 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:08:37.972 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T04:08:38.239 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:08:38.239 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T04:08:38.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:08:38.338 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T04:08:38.387 INFO:tasks.ceph:Archiving mon data... 2024-09-22T04:08:38.388 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406/data/mon.a.tgz 2024-09-22T04:08:38.389 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2024-09-22T04:08:38.404 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.tMCBqLbj13 2024-09-22T04:08:38.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.tMCBqLbj13 2024-09-22T04:08:38.541 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.tMCBqLbj13 2024-09-22T04:08:38.643 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.tMCBqLbj13 is 202KB 2024-09-22T04:08:38.696 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.tMCBqLbj13 2024-09-22T04:08:38.712 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T04:08:38.712 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T04:08:38.839 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T04:08:38.839 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T04:08:38.840 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406/remote/smithi052/crash 2024-09-22T04:08:38.841 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T04:08:38.904 INFO:tasks.ceph:Compressing logs... 2024-09-22T04:08:38.904 DEBUG:teuthology.orchestra.run.smithi052:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T04:08:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T04:08:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29873.log 2024-09-22T04:08:38.978 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30335.log 2024-09-22T04:08:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29661.log 2024-09-22T04:08:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30125.log 2024-09-22T04:08:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30039.log 2024-09-22T04:08:38.979 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30419.log 2024-09-22T04:08:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T04:08:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28765.log 2024-09-22T04:08:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29383.log 2024-09-22T04:08:38.980 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29124.log 2024-09-22T04:08:38.981 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29154.log 2024-09-22T04:08:38.981 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28915.log 2024-09-22T04:08:38.981 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28677.log 2024-09-22T04:08:38.981 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T04:08:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29579.log 2024-09-22T04:08:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29660.log 2024-09-22T04:08:38.982 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31556.log 2024-09-22T04:08:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-22T04:08:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30145.log 2024-09-22T04:08:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29659.log 2024-09-22T04:08:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27722.log 2024-09-22T04:08:38.983 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30038.log 2024-09-22T04:08:38.984 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29785.log 2024-09-22T04:08:38.984 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29912.log 2024-09-22T04:08:38.984 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-09-22T04:08:38.984 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T04:08:38.984 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28959.log 2024-09-22T04:08:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30251.log 2024-09-22T04:08:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29999.log 2024-09-22T04:08:38.985 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29453.log 2024-09-22T04:08:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T04:08:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31907.log 2024-09-22T04:08:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29300.log 2024-09-22T04:08:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29787.log 2024-09-22T04:08:38.986 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29913.log 2024-09-22T04:08:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T04:08:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29786.log 2024-09-22T04:08:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30271.log 2024-09-22T04:08:38.987 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T04:08:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31605.log 2024-09-22T04:08:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30166.log 2024-09-22T04:08:38.988 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28875.log 2024-09-22T04:08:38.993 INFO:teuthology.orchestra.run.smithi052.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29537.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.30335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31521.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.31556.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.29537.log: /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.31907.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30377.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.29873.log: /var/log/ceph/ceph-client.admin.30125.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31907.log.gz 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.30039.log: /var/log/ceph/ceph-client.admin.28765.log: /var/log/ceph/ceph-client.admin.29661.log: /var/log/ceph/ceph-client.admin.30419.log: /var/log/ceph/ceph-client.admin.31521.log: /var/log/ceph/ceph-client.admin.29356.log: /var/log/ceph/ceph-client.admin.29659.log: /var/log/ceph/ceph-client.admin.31456.log: gzip/var/log/ceph/ceph-client.admin.30145.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31582.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr: 84.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.29383.log: /var/log/ceph/ceph-client.admin.29912.log: /var/log/ceph/ceph-client.admin.30038.log: /var/log/ceph/ceph-client.admin.29300.log: /var/log/ceph/ceph-client.admin.29999.log: /var/log/ceph/ceph.tmp-client.admin.27722.log: /var/log/ceph/ceph-client.admin.30377.log: /var/log/ceph/ceph-client.admin.29154.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29124.log: /var/log/ceph/ceph-client.admin.29495.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.28959.log: /var/log/ceph/ceph-client.admin.29453.log: /var/log/ceph/ceph-client.admin.29913.log: /var/log/ceph/ceph-client.admin.29785.log: /var/log/ceph/ceph-client.admin.29786.log: /var/log/ceph/ceph-client.admin.28915.log: /var/log/ceph/ceph-client.admin.28677.log: /var/log/ceph/ceph-client.admin.29787.log: /var/log/ceph/ceph-client.admin.30271.log: /var/log/ceph/ceph-client.admin.30166.log: /var/log/ceph/ceph-client.admin.30251.log: /var/log/ceph/ceph-client.admin.29579.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29660.log: /var/log/ceph/ceph-client.admin.31669.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.28875.log: /var/log/ceph/ceph-client.admin.31605.log: /var/log/ceph/ceph-client.admin.31582.log: /var/log/ceph/ceph-client.admin.29495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31670.log 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.31669.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.31669.log.gz 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr:/var/log/ceph/ceph-client.admin.31670.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.31670.log.gz 2024-09-22T04:08:38.994 INFO:teuthology.orchestra.run.smithi052.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T04:08:39.001 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28765.log.gz 2024-09-22T04:08:39.004 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30335.log.gz 2024-09-22T04:08:39.004 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30377.log.gz 2024-09-22T04:08:39.004 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30419.log.gz 2024-09-22T04:08:39.005 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28677.log.gz 2024-09-22T04:08:39.011 INFO:teuthology.orchestra.run.smithi052.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28959.log.gz 2024-09-22T04:08:39.019 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28915.log.gz 2024-09-22T04:08:39.021 INFO:teuthology.orchestra.run.smithi052.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T04:08:39.026 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28875.log.gz 2024-09-22T04:08:39.028 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29154.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29300.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.29383.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29453.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29495.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29537.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29579.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29659.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29661.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29787.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29786.log.gz 2024-09-22T04:08:39.039 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29785.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29660.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29873.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29913.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29912.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29999.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30038.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30039.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.30125.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30145.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30166.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30271.log.gz -- replaced with /var/log/ceph/ceph-client.admin.30251.log.gz 2024-09-22T04:08:39.040 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:08:39.041 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-09-22T04:08:39.041 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31521.log.gz 2024-09-22T04:08:39.041 INFO:teuthology.orchestra.run.smithi052.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31556.log.gz 2024-09-22T04:08:39.042 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31605.log.gz 2024-09-22T04:08:39.042 INFO:teuthology.orchestra.run.smithi052.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31582.log.gz 2024-09-22T04:08:39.043 INFO:teuthology.orchestra.run.smithi052.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27722.log.gz 2024-09-22T04:08:39.049 INFO:teuthology.orchestra.run.smithi052.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29124.log.gz 2024-09-22T04:08:39.182 INFO:teuthology.orchestra.run.smithi052.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T04:08:41.907 INFO:teuthology.orchestra.run.smithi052.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T04:08:42.050 INFO:teuthology.orchestra.run.smithi052.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T04:08:42.381 INFO:teuthology.orchestra.run.smithi052.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-22T04:08:42.383 INFO:teuthology.orchestra.run.smithi052.stderr: 2024-09-22T04:08:42.383 INFO:teuthology.orchestra.run.smithi052.stderr:real 0m3.423s 2024-09-22T04:08:42.383 INFO:teuthology.orchestra.run.smithi052.stderr:user 0m4.504s 2024-09-22T04:08:42.383 INFO:teuthology.orchestra.run.smithi052.stderr:sys 0m0.186s 2024-09-22T04:08:42.383 INFO:tasks.ceph:Archiving logs... 2024-09-22T04:08:42.383 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406/remote/smithi052/log 2024-09-22T04:08:42.384 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T04:08:42.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T04:08:42.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T04:08:42.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-22T04:08:42.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-22T04:08:43.333 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T04:08:43.341 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T04:08:43.341 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T04:08:43.399 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T04:08:43.408 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T04:08:43.408 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T04:08:43.429 INFO:teuthology.orchestra.run.smithi052.stderr:bash: line 1: ntpq: command not found 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:=============================================================================== 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 -489us[ -488us] +/- 50ms 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 21 +982us[ +982us] +/- 59ms 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -476us[ -475us] +/- 65ms 2024-09-22T04:08:43.496 INFO:teuthology.orchestra.run.smithi052.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T04:08:43.517 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T04:08:43.526 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T04:08:43.527 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T04:08:43.536 DEBUG:teuthology.orchestra.run.smithi052:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T04:08:43.676 DEBUG:teuthology.orchestra.run.smithi052:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-22T04:08:43.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T04:08:43.704 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T04:08:43.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T04:08:43.725 INFO:teuthology.task.internal:Duration was 580.550821 seconds 2024-09-22T04:08:43.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T04:08:43.733 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T04:08:43.734 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T04:08:43.869 INFO:teuthology.orchestra.run.smithi052.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T04:08:44.002 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T04:08:44.002 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2024-09-22T04:08:44.002 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T04:08:44.023 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T04:08:44.023 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T04:08:44.090 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T04:08:44.090 DEBUG:teuthology.orchestra.run.smithi052:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T04:08:44.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T04:08:44.258 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T04:08:44.258 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T04:08:44.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T04:08:44.329 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T04:08:44.370 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2024-09-22T04:08:44.398 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T04:08:44.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T04:08:44.452 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T04:08:44.462 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T04:08:44.462 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_20:56:02-crimson-rados-main-distro-crimson-smithi/7914406/remote/smithi052 2024-09-22T04:08:44.462 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T04:08:44.581 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T04:08:44.581 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T04:08:44.619 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T04:08:44.630 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T04:08:44.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T04:08:44.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T04:08:44.639 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T04:08:44.674 INFO:teuthology.orchestra.run.smithi052.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 04:08 /home/ubuntu/cephtest 2024-09-22T04:08:44.675 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T04:08:44.684 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T04:08:44.703 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=main&var-job_id_selected=7914406 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench} duration: 580.5508213043213 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T04:08:44.704 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T04:08:44.779 INFO:teuthology.run:pass