2024-09-22T03:25:43.220 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-22T03:25:43.240 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:25:43.329 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785 branch: squid description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914785' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 osd objectstore: seastore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid env: CRIMSON_COMPAT: '1' sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1158 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 0/1 suite: crimson-rados suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS2s2o+lsmnwUlZ00b5vyZQeHztirg0JS2Z8OGAUPRSkb5Yo9neStTCLsmdnDsbziSrD/COEboD8/j+AlEwl7E= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:56:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-22T03:25:43.329 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-22T03:25:43.330 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-22T03:25:43.330 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-22T03:25:43.332 INFO:teuthology.task.internal:Checking packages... 2024-09-22T03:25:43.356 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-22T03:25:43.356 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-22T03:25:43.356 INFO:teuthology.packaging:ref: None 2024-09-22T03:25:43.356 INFO:teuthology.packaging:tag: None 2024-09-22T03:25:43.356 INFO:teuthology.packaging:branch: squid 2024-09-22T03:25:43.356 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:25:43.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-22T03:25:43.575 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-22T03:25:43.576 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-22T03:25:43.582 INFO:teuthology.task.internal:no buildpackages task found 2024-09-22T03:25:43.582 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-22T03:25:43.610 INFO:teuthology.task.internal:Saving configuration 2024-09-22T03:25:43.625 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-22T03:25:43.632 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-22T03:25:43.654 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-22 03:20:53.001233', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS2s2o+lsmnwUlZ00b5vyZQeHztirg0JS2Z8OGAUPRSkb5Yo9neStTCLsmdnDsbziSrD/COEboD8/j+AlEwl7E='} 2024-09-22T03:25:43.654 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-22T03:25:43.660 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-22T03:25:43.660 INFO:teuthology.run_tasks:Running task console_log... 2024-09-22T03:25:43.692 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f38af25c160>, signals=[15]) 2024-09-22T03:25:43.693 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-22T03:25:43.699 INFO:teuthology.task.internal:Opening connections... 2024-09-22T03:25:43.699 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:25:43.700 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:25:43.773 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-22T03:25:43.780 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-09-22T03:25:43.852 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-09-22T03:25:43.852 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-09-22T03:25:43.909 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-09-22T03:25:43.909 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-09-22T03:25:43.910 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-09-22T03:25:43.910 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-09-22T03:25:43.910 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-09-22T03:25:43.910 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-09-22T03:25:43.910 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-22T03:25:43.911 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-22T03:25:43.912 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-09-22T03:25:43.930 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-22T03:25:43.938 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-22T03:25:43.942 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-22T03:25:43.943 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-09-22T03:25:43.967 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-22T03:25:43.972 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-22T03:25:43.973 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-09-22T03:25:44.126 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-22T03:25:44.158 INFO:teuthology.run_tasks:Running task kernel... 2024-09-22T03:25:44.171 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-22T03:25:44.172 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-22T03:25:44.172 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-22T03:25:44.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:25:44.190 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-22T03:25:44.248 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-503.el9.x86_64 2024-09-22T03:25:44.248 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-503.el9.x86_64 2024-09-22T03:25:44.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-09-22T03:25:47.601 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-22T03:25:51.926 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 20 MB 00:03 2024-09-22T03:25:57.268 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.5 MB 00:01 2024-09-22T03:25:58.843 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 19 kB 00:00 2024-09-22T03:26:01.497 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 8.9 MB/s | 23 MB 00:02 2024-09-22T03:26:06.952 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-22T03:26:08.842 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-22T03:26:08.842 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-22T03:26:08.936 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:26:08.947 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:26:08.947 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-09-22T03:26:08.947 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:26:08.947 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-09-22T03:26:08.948 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:26:08.949 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 79 M 2024-09-22T03:26:08.949 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 126 M 2024-09-22T03:26:08.949 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:26:09.353 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 289 kB/s | 36 kB 00:00 2024-09-22T03:26:11.920 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 5.8 MB/s | 16 MB 00:02 2024-09-22T03:26:14.461 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 5.6 MB/s | 29 MB 00:05 2024-09-22T03:26:15.770 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 5.3 MB/s | 35 MB 00:06 2024-09-22T03:26:15.770 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:26:15.770 INFO:teuthology.orchestra.run.smithi117.stdout:Total 12 MB/s | 79 MB 00:06 2024-09-22T03:26:16.456 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:26:16.823 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:26:16.823 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:26:17.780 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:26:17.780 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:26:20.205 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:26:20.892 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:26:20.980 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:26:23.734 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:26:23.865 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:26:28.723 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:26:28.864 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:26:33.707 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:27:11.919 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:27:11.958 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:27:12.329 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:27:12.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-22T03:27:12.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-22T03:27:12.329 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:27:13.163 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:27:13.526 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:27:13.623 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.623 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:27:13.623 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-22T03:27:13.624 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-22T03:27:13.624 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-22T03:27:13.624 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-22T03:27:13.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-09-22T03:27:13.624 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-09-22T03:27:13.682 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-22T03:27:13.683 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-22T03:27:13.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 82.el9 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-09-22T03:27:13.715 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8274670 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-22T03:27:13.716 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-22T03:27:13.717 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-22T03:27:13.717 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-09-22T03:27:13.717 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:27:13.717 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-09-22T03:27:13.718 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-22T03:27:13.718 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-508.el9.x86_64 2024-09-22T03:27:13.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-22T03:27:14.379 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-09-22T03:27:15.743 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-22T03:27:15.769 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-09-22T03:27:15.772 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-22T03:27:15.787 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.duJRMfDCRG 2024-09-22T03:27:15.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.duJRMfDCRG 2024-09-22T03:27:15.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.duJRMfDCRG 2024-09-22T03:27:15.990 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.duJRMfDCRG is 6KB 2024-09-22T03:27:16.010 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.duJRMfDCRG 2024-09-22T03:27:16.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-09-22T03:27:16.094 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-22T03:27:16.094 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-22T03:27:16.094 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-22T03:27:16.094 INFO:teuthology.orchestra.run.smithi117.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-22T03:27:16.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-22T03:27:16.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-09-22T03:27:46.273 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-22T03:27:46.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-22T03:27:46.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:28:46.335 DEBUG:teuthology.orchestra.remote:timed out 2024-09-22T03:28:55.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-22T03:28:55.337 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:28:55.675 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-22T03:28:56.061 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-22T03:28:56.062 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-22T03:28:56.062 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-09-22T03:28:56.077 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-508.el9.x86_64 2024-09-22T03:28:56.077 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-22T03:28:56.077 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-22T03:28:56.077 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-22T03:28:57.078 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-22T03:28:57.078 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-22T03:28:57.164 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-09-22T03:28:57.186 DEBUG:teuthology.parallel:result is None 2024-09-22T03:28:57.186 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-22T03:28:57.193 INFO:teuthology.task.internal:Creating test directory... 2024-09-22T03:28:57.193 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-22T03:28:57.207 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-22T03:28:57.213 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-22T03:28:57.222 INFO:teuthology.task.internal:Creating archive directory... 2024-09-22T03:28:57.223 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-22T03:28:57.264 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-22T03:28:57.270 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-22T03:28:57.270 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-22T03:28:57.340 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:28:57.350 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-22T03:28:57.352 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-22T03:28:57.359 INFO:teuthology.task.internal:Configuring sudo... 2024-09-22T03:28:57.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-22T03:28:57.383 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-22T03:28:57.391 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-22T03:28:57.391 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-22T03:28:57.435 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:28:57.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-22T03:28:57.559 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:28:57.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-22T03:28:57.674 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:28:57.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-22T03:28:57.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-09-22T03:28:57.823 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:28:57.958 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-22T03:28:57.966 INFO:teuthology.task.internal:Starting timer... 2024-09-22T03:28:57.966 INFO:teuthology.run_tasks:Running task pcp... 2024-09-22T03:28:58.012 INFO:teuthology.run_tasks:Running task selinux... 2024-09-22T03:28:58.049 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-22T03:28:58.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-09-22T03:28:58.097 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-09-22T03:28:58.099 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-22T03:28:58.099 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-09-22T03:28:58.123 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-09-22T03:28:58.123 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-09-22T03:28:58.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:28:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:28:58.184 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-22T03:28:58.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-09-22T03:28:58.262 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-22T03:28:58.273 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-22T03:28:58.284 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-22T03:28:58.285 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-22T03:32:41.281 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2024-09-22T03:32:41.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-22T03:32:41.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-22T03:32:41.358 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-09-22T03:32:41.439 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-09-22T03:32:41.440 INFO:teuthology.run_tasks:Running task clock... 2024-09-22T03:32:41.453 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-22T03:32:41.454 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-22T03:32:41.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:32:41.510 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-22T03:32:41.525 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-22T03:32:41.551 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-09-22T03:32:41.563 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-09-22T03:32:41.577 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-22T03:32:41.591 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-22T03:32:41.644 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:32:41.648 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:32:41.649 INFO:teuthology.run_tasks:Running task install... 2024-09-22T03:32:41.657 DEBUG:teuthology.task.install:project ceph 2024-09-22T03:32:41.657 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-22T03:32:41.657 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-22T03:32:41.657 INFO:teuthology.task.install:Using flavor: crimson 2024-09-22T03:32:41.662 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-22T03:32:41.662 INFO:teuthology.task.install:extra packages: [] 2024-09-22T03:32:41.665 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'tag': None, 'wait_for_package': False} 2024-09-22T03:32:41.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:32:41.811 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/ 2024-09-22T03:32:41.811 INFO:teuthology.task.install.rpm:Package version is 19.1.1-373.g4e45e6b4 2024-09-22T03:32:41.921 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-22T03:32:41.921 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:32:41.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-22T03:32:41.948 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-22T03:32:41.948 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e45e6b490e6cbe082cdf155fea780438ec53fc7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-22T03:32:42.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-22T03:32:42.089 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-22T03:32:42.153 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-09-22T03:32:42.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-09-22T03:32:42.381 INFO:teuthology.orchestra.run.smithi117.stdout:45 files removed 2024-09-22T03:32:42.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-09-22T03:32:42.974 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 311 kB/s | 87 kB 00:00 2024-09-22T03:32:43.267 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-22T03:32:43.550 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 8.0 kB/s | 2.0 kB 00:00 2024-09-22T03:32:44.067 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-22T03:32:47.369 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-22T03:32:51.710 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-22T03:32:53.469 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-22T03:32:55.490 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 12 MB/s | 23 MB 00:01 2024-09-22T03:33:00.912 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-22T03:33:03.105 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 9.9 M 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.2 M 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:33:03.138 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.4 M 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 19 M 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 25 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 760 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 501 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 5.1 M 2024-09-22T03:33:03.139 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 45 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 136 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 161 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 321 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 301 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 99 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-22T03:33:03.140 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-09-22T03:33:03.141 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:33:04.538 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.1.1-373.g4e45e6b4.el9.x86_ 5.2 MB/s | 5.4 MB 00:01 2024-09-22T03:33:04.588 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.1.1-373.g4e45e6b4.el9.x 503 kB/s | 25 kB 00:00 2024-09-22T03:33:04.764 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): ceph-radosgw-19.1.1-373.g4e45e6b4.el9.x 7.8 MB/s | 9.9 MB 00:01 2024-09-22T03:33:04.789 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libcephfs2-19.1.1-373.g4e45e6b4.el9.x86 3.7 MB/s | 760 kB 00:00 2024-09-22T03:33:04.831 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): libradosstriper1-19.1.1-373.g4e45e6b4.e 7.4 MB/s | 501 kB 00:00 2024-09-22T03:33:04.897 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): python3-ceph-argparse-19.1.1-373.g4e45e 679 kB/s | 45 kB 00:00 2024-09-22T03:33:04.956 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-common-19.1.1-373.g4e45e6b 2.3 MB/s | 136 kB 00:00 2024-09-22T03:33:05.032 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-cephfs-19.1.1-373.g4e45e6b4.el9 2.1 MB/s | 161 kB 00:00 2024-09-22T03:33:05.123 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): librgw2-19.1.1-373.g4e45e6b4.el9.x86_64 15 MB/s | 5.1 MB 00:00 2024-09-22T03:33:05.149 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rados-19.1.1-373.g4e45e6b4.el9 2.7 MB/s | 321 kB 00:00 2024-09-22T03:33:05.182 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rbd-19.1.1-373.g4e45e6b4.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-22T03:33:05.208 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): python3-rgw-19.1.1-373.g4e45e6b4.el9.x 1.6 MB/s | 99 kB 00:00 2024-09-22T03:33:05.858 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 158 kB/s | 107 kB 00:00 2024-09-22T03:33:05.908 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 229 kB/s | 160 kB 00:00 2024-09-22T03:33:06.292 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): ceph-common-19.1.1-373.g4e45e6b4.el9.x 7.0 MB/s | 19 MB 00:02 2024-09-22T03:33:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-09-22T03:33:07.059 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 576 kB/s | 662 kB 00:01 2024-09-22T03:33:07.176 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 334 kB/s | 292 kB 00:00 2024-09-22T03:33:07.210 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 67 kB/s | 42 kB 00:00 2024-09-22T03:33:07.735 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 64 kB/s | 36 kB 00:00 2024-09-22T03:33:07.794 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 5.2 MB/s | 308 kB 00:00 2024-09-22T03:33:07.819 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 78 kB/s | 47 kB 00:00 2024-09-22T03:33:07.852 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): qatlib-24.02.0-1.el9.x86_64.rpm 278 kB/s | 221 kB 00:00 2024-09-22T03:33:07.878 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-09-22T03:33:07.970 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 422 kB/s | 49 kB 00:00 2024-09-22T03:33:08.095 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): libarrow-9.0.0-11.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-09-22T03:33:08.120 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.6 MB/s | 191 kB 00:00 2024-09-22T03:33:08.162 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-09-22T03:33:08.229 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-22T03:33:08.413 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): librados2-19.1.1-373.g4e45e6b4.el9.x86 13 MB/s | 3.2 MB 00:00 2024-09-22T03:33:08.438 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): libunwind-1.6.2-1.el9.x86_64.rpm 122 kB/s | 67 kB 00:00 2024-09-22T03:33:08.563 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): librbd1-19.1.1-373.g4e45e6b4.el9.x86_6 9.3 MB/s | 3.1 MB 00:00 2024-09-22T03:33:08.568 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:08.568 INFO:teuthology.orchestra.run.smithi117.stdout:Total 11 MB/s | 58 MB 00:05 2024-09-22T03:33:08.672 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:33:08.709 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:33:08.709 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:33:09.370 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:33:09.370 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:33:10.848 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:33:11.273 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-22T03:33:11.332 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:33:11.423 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:33:11.514 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:33:11.550 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:33:11.629 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:33:11.662 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/34 2024-09-22T03:33:11.688 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/34 2024-09-22T03:33:11.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:33:11.734 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 7/34 2024-09-22T03:33:12.147 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:33:12.174 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 8/34 2024-09-22T03:33:12.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/34 2024-09-22T03:33:12.247 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 10/34 2024-09-22T03:33:12.276 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:33:12.353 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 12/34 2024-09-22T03:33:12.381 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:33:12.480 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:33:12.522 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 14/34 2024-09-22T03:33:12.635 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-22T03:33:12.673 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-22T03:33:12.723 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-22T03:33:12.755 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-22T03:33:12.784 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-22T03:33:12.817 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-22T03:33:13.152 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-22T03:33:13.204 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:33:13.295 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 22/34 2024-09-22T03:33:13.325 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/34 2024-09-22T03:33:13.362 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-22T03:33:13.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-22T03:33:13.881 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:33:13.953 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 26/34 2024-09-22T03:33:14.031 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 27/34 2024-09-22T03:33:14.067 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 28/34 2024-09-22T03:33:16.301 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:33:16.338 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:33:16.912 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 29/34 2024-09-22T03:33:16.950 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:33:17.055 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 30/34 2024-09-22T03:33:17.055 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-22T03:33:17.055 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-22T03:33:17.056 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:17.093 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:33:27.016 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:33:27.016 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-09-22T03:33:27.016 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-09-22T03:33:27.016 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-09-22T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-09-22T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-09-22T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-09-22T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-09-22T03:33:27.017 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:28.010 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 32/34 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-22T03:33:28.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:28.037 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:33:28.130 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-22T03:33:28.130 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x 8/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86 9/34 2024-09-22T03:33:29.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 10/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 11/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 12/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 13/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-22T03:33:29.319 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-22T03:33:29.320 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 31/34 2024-09-22T03:33:29.321 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-22T03:33:29.321 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 33/34 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.337 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.338 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-22T03:33:30.339 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-20.el9.x86_64 2024-09-22T03:33:30.340 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-22T03:33:30.340 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:30.340 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:33:30.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-09-22T03:33:31.291 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:31 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:33:31.422 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:33:31.423 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:31.425 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:33:31.425 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:31.425 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:33:31.425 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 124 M 2024-09-22T03:33:31.425 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: libasan x86_64 11.5.0-2.el9 appstream 414 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 158 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: libubsan x86_64 11.5.0-2.el9 appstream 183 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout:Install 7 Packages 2024-09-22T03:33:31.426 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:33:31.427 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 125 M 2024-09-22T03:33:31.427 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.1 G 2024-09-22T03:33:31.427 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:33:31.968 INFO:teuthology.orchestra.run.smithi117.stdout:(1/7): libcephsqlite-19.1.1-373.g4e45e6b4.el9.x 672 kB/s | 158 kB 00:00 2024-09-22T03:33:32.468 INFO:teuthology.orchestra.run.smithi117.stdout:(2/7): libasan-11.5.0-2.el9.x86_64.rpm 564 kB/s | 414 kB 00:00 2024-09-22T03:33:32.518 INFO:teuthology.orchestra.run.smithi117.stdout:(3/7): libubsan-11.5.0-2.el9.x86_64.rpm 332 kB/s | 183 kB 00:00 2024-09-22T03:33:32.593 INFO:teuthology.orchestra.run.smithi117.stdout:(4/7): socat-1.7.4.1-6.el9.x86_64.rpm 2.4 MB/s | 304 kB 00:00 2024-09-22T03:33:32.627 INFO:teuthology.orchestra.run.smithi117.stdout:(5/7): xmlstarlet-1.6.1-20.el9.x86_64.rpm 587 kB/s | 64 kB 00:00 2024-09-22T03:33:32.660 INFO:teuthology.orchestra.run.smithi117.stdout:(6/7): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.8 MB/s | 122 kB 00:00 2024-09-22T03:33:35.152 INFO:teuthology.orchestra.run.smithi117.stdout:(7/7): ceph-test-19.1.1-373.g4e45e6b4.el9.x86_6 36 MB/s | 124 MB 00:03 2024-09-22T03:33:35.155 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:33:35.155 INFO:teuthology.orchestra.run.smithi117.stdout:Total 34 MB/s | 125 MB 00:03 2024-09-22T03:33:35.207 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:33:35.228 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:33:35.228 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:33:36.046 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:33:36.046 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:33:37.108 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:33:37.188 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/7 2024-09-22T03:33:37.236 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/7 2024-09-22T03:33:37.279 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 3/7 2024-09-22T03:33:37.311 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:33:37.361 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libasan-11.5.0-2.el9.x86_64 5/7 2024-09-22T03:33:37.389 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:33:54.637 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 6/7 2024-09-22T03:33:54.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:34:02.918 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 7/7 2024-09-22T03:34:02.918 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/7 2024-09-22T03:34:02.918 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/7 2024-09-22T03:34:02.919 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libasan-11.5.0-2.el9.x86_64 3/7 2024-09-22T03:34:02.919 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libubsan-11.5.0-2.el9.x86_64 4/7 2024-09-22T03:34:02.919 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 5/7 2024-09-22T03:34:02.919 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/7 2024-09-22T03:34:03.266 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 7/7 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: libasan-11.5.0-2.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: libubsan-11.5.0-2.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:03.267 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:03.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-09-22T03:34:04.167 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:04 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:04.306 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 6.6 k 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 2.3 M 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 1.5 M 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 249 k 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 4.9 M 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 120 M 2024-09-22T03:34:04.309 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-22T03:34:04.310 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-22T03:34:04.311 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout:Install 38 Packages 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:04.312 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 134 M 2024-09-22T03:34:04.313 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.4 G 2024-09-22T03:34:04.313 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:34:04.764 INFO:teuthology.orchestra.run.smithi117.stdout:(1/38): ceph-19.1.1-373.g4e45e6b4.el9.x86_64.rp 19 kB/s | 6.6 kB 00:00 2024-09-22T03:34:05.522 INFO:teuthology.orchestra.run.smithi117.stdout:(2/38): ceph-mgr-19.1.1-373.g4e45e6b4.el9.x86_6 1.4 MB/s | 1.5 MB 00:01 2024-09-22T03:34:05.831 INFO:teuthology.orchestra.run.smithi117.stdout:(3/38): ceph-mds-19.1.1-373.g4e45e6b4.el9.x86_6 1.6 MB/s | 2.3 MB 00:01 2024-09-22T03:34:06.039 INFO:teuthology.orchestra.run.smithi117.stdout:(4/38): ceph-mgr-modules-core-19.1.1-373.g4e45e 1.2 MB/s | 249 kB 00:00 2024-09-22T03:34:06.156 INFO:teuthology.orchestra.run.smithi117.stdout:(5/38): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-09-22T03:34:06.265 INFO:teuthology.orchestra.run.smithi117.stdout:(6/38): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-22T03:34:06.336 INFO:teuthology.orchestra.run.smithi117.stdout:(7/38): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 106 kB 00:00 2024-09-22T03:34:06.416 INFO:teuthology.orchestra.run.smithi117.stdout:(8/38): python3-pycparser-2.20-6.el9.noarch.rpm 1.7 MB/s | 135 kB 00:00 2024-09-22T03:34:06.483 INFO:teuthology.orchestra.run.smithi117.stdout:(9/38): python3-requests-2.25.1-8.el9.noarch.rp 1.8 MB/s | 125 kB 00:00 2024-09-22T03:34:06.550 INFO:teuthology.orchestra.run.smithi117.stdout:(10/38): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-22T03:34:06.683 INFO:teuthology.orchestra.run.smithi117.stdout:(11/38): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-09-22T03:34:06.909 INFO:teuthology.orchestra.run.smithi117.stdout:(12/38): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-09-22T03:34:06.976 INFO:teuthology.orchestra.run.smithi117.stdout:(13/38): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-09-22T03:34:07.009 INFO:teuthology.orchestra.run.smithi117.stdout:(14/38): python3-markupsafe-1.1.1-12.el9.x86_64 1.0 MB/s | 35 kB 00:00 2024-09-22T03:34:07.068 INFO:teuthology.orchestra.run.smithi117.stdout:(15/38): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-22T03:34:07.101 INFO:teuthology.orchestra.run.smithi117.stdout:(16/38): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-09-22T03:34:07.135 INFO:teuthology.orchestra.run.smithi117.stdout:(17/38): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-22T03:34:07.194 INFO:teuthology.orchestra.run.smithi117.stdout:(18/38): lua-devel-5.4.4-4.el9.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-09-22T03:34:07.261 INFO:teuthology.orchestra.run.smithi117.stdout:(19/38): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-22T03:34:07.286 INFO:teuthology.orchestra.run.smithi117.stdout:(20/38): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-22T03:34:07.328 INFO:teuthology.orchestra.run.smithi117.stdout:(21/38): python3-cheroot-8.6.0-4.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-09-22T03:34:07.395 INFO:teuthology.orchestra.run.smithi117.stdout:(22/38): ceph-mon-19.1.1-373.g4e45e6b4.el9.x86_ 1.9 MB/s | 4.9 MB 00:02 2024-09-22T03:34:07.445 INFO:teuthology.orchestra.run.smithi117.stdout:(23/38): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-22T03:34:07.470 INFO:teuthology.orchestra.run.smithi117.stdout:(24/38): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-09-22T03:34:07.496 INFO:teuthology.orchestra.run.smithi117.stdout:(25/38): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-22T03:34:07.521 INFO:teuthology.orchestra.run.smithi117.stdout:(26/38): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-22T03:34:07.546 INFO:teuthology.orchestra.run.smithi117.stdout:(27/38): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-22T03:34:07.571 INFO:teuthology.orchestra.run.smithi117.stdout:(28/38): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-22T03:34:07.597 INFO:teuthology.orchestra.run.smithi117.stdout:(29/38): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-22T03:34:07.622 INFO:teuthology.orchestra.run.smithi117.stdout:(30/38): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-22T03:34:07.656 INFO:teuthology.orchestra.run.smithi117.stdout:(31/38): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-09-22T03:34:07.681 INFO:teuthology.orchestra.run.smithi117.stdout:(32/38): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-22T03:34:07.706 INFO:teuthology.orchestra.run.smithi117.stdout:(33/38): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-22T03:34:07.740 INFO:teuthology.orchestra.run.smithi117.stdout:(34/38): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-22T03:34:07.774 INFO:teuthology.orchestra.run.smithi117.stdout:(35/38): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-22T03:34:07.799 INFO:teuthology.orchestra.run.smithi117.stdout:(36/38): python3-zc-lockfile-2.0-10.el9.noarch. 798 kB/s | 20 kB 00:00 2024-09-22T03:34:07.824 INFO:teuthology.orchestra.run.smithi117.stdout:(37/38): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-22T03:34:15.732 INFO:teuthology.orchestra.run.smithi117.stdout:(38/38): ceph-osd-19.1.1-373.g4e45e6b4.el9.x86_ 12 MB/s | 120 MB 00:10 2024-09-22T03:34:15.737 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:15.737 INFO:teuthology.orchestra.run.smithi117.stdout:Total 12 MB/s | 134 MB 00:11 2024-09-22T03:34:15.911 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:34:15.952 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:34:15.952 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:34:17.065 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:34:17.066 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:34:19.067 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:34:19.117 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/38 2024-09-22T03:34:19.157 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/38 2024-09-22T03:34:19.204 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/38 2024-09-22T03:34:19.273 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/38 2024-09-22T03:34:19.344 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/38 2024-09-22T03:34:19.385 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/38 2024-09-22T03:34:19.484 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/38 2024-09-22T03:34:19.555 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/38 2024-09-22T03:34:19.595 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/38 2024-09-22T03:34:19.648 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/38 2024-09-22T03:34:19.678 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/38 2024-09-22T03:34:19.713 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/38 2024-09-22T03:34:19.746 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/38 2024-09-22T03:34:19.787 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/38 2024-09-22T03:34:19.822 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/38 2024-09-22T03:34:19.864 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/38 2024-09-22T03:34:19.921 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/38 2024-09-22T03:34:19.961 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/38 2024-09-22T03:34:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/38 2024-09-22T03:34:20.172 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/38 2024-09-22T03:34:20.225 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/38 2024-09-22T03:34:39.105 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 22/38 2024-09-22T03:34:39.138 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:34:39.167 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 23/38 2024-09-22T03:34:39.167 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:39.168 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-22T03:34:39.168 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:34:39.168 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-22T03:34:39.168 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:39.278 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 24/38 2024-09-22T03:34:39.326 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 25/38 2024-09-22T03:34:39.374 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 26/38 2024-09-22T03:34:39.430 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 27/38 2024-09-22T03:34:39.614 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 28/38 2024-09-22T03:34:39.663 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 29/38 2024-09-22T03:34:39.742 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/38 2024-09-22T03:34:39.831 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 31/38 2024-09-22T03:34:39.864 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 32/38 2024-09-22T03:34:39.980 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 33/38 2024-09-22T03:34:40.139 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 34/38 2024-09-22T03:34:40.167 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:34:40.190 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 35/38 2024-09-22T03:34:40.190 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:40.191 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-22T03:34:40.191 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:34:40.191 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-22T03:34:40.191 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:40.659 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 36/38 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-22T03:34:40.683 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:40.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:34:40.942 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 37/38 2024-09-22T03:34:40.942 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:40.942 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-22T03:34:40.942 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:34:40.943 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-22T03:34:40.943 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:41.006 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:34:50.928 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 38/38 2024-09-22T03:34:50.928 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/38 2024-09-22T03:34:50.928 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 5/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.n 6/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 14/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 15/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 16/38 2024-09-22T03:34:50.929 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 17/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 18/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 19/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 20/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 21/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 22/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 23/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 24/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 25/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 26/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 27/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 28/38 2024-09-22T03:34:50.930 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 29/38 2024-09-22T03:34:50.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 30/38 2024-09-22T03:34:50.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 31/38 2024-09-22T03:34:50.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 32/38 2024-09-22T03:34:50.931 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 33/38 2024-09-22T03:34:50.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 34/38 2024-09-22T03:34:50.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 35/38 2024-09-22T03:34:50.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 36/38 2024-09-22T03:34:50.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 37/38 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 38/38 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-09-22T03:34:51.945 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-22T03:34:51.946 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-22T03:34:51.947 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:51.948 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:52.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-09-22T03:34:52.778 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:52 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:52.867 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:52.909 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:52.910 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:34:52.910 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:52.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-09-22T03:34:53.556 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:53 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:53.686 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 765 k 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:34:53.687 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:53.688 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 765 k 2024-09-22T03:34:53.688 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 771 k 2024-09-22T03:34:53.688 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:34:53.965 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.1.1-373.g4e45e6b4.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-22T03:34:53.966 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:53.966 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-09-22T03:34:53.966 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:34:53.971 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:34:53.971 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:34:53.977 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:34:53.977 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:34:54.052 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:34:54.467 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:54.561 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:54.903 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:55.161 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:34:55.162 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:55.162 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:34:55.162 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:34:55.162 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:55.162 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:55.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-09-22T03:34:55.984 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:55 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:56.119 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:56.119 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:56.119 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 147 k 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 147 k 2024-09-22T03:34:56.120 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 481 k 2024-09-22T03:34:56.121 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:34:56.337 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.1.1-373.g4e45e6b 680 kB/s | 147 kB 00:00 2024-09-22T03:34:56.338 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:34:56.338 INFO:teuthology.orchestra.run.smithi117.stdout:Total 675 kB/s | 147 kB 00:00 2024-09-22T03:34:56.338 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:34:56.345 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:34:56.346 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:34:56.402 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:34:56.403 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:34:56.554 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:34:56.622 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:34:56.647 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:34:56.647 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:34:56.647 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-22T03:34:56.647 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.e 1/1 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:57.569 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:57.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-09-22T03:34:58.399 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:58 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:58.490 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:34:58.532 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:58.533 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:34:58.534 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:34:58.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-09-22T03:34:59.190 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:59 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:34:59.321 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 3.6 M 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 32 k 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 16 k 2024-09-22T03:34:59.322 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:34:59.323 INFO:teuthology.orchestra.run.smithi117.stdout:Install 11 Packages 2024-09-22T03:34:59.324 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:34:59.324 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 7.7 M 2024-09-22T03:34:59.324 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 101 M 2024-09-22T03:34:59.324 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:34:59.791 INFO:teuthology.orchestra.run.smithi117.stdout:(1/11): ceph-prometheus-alerts-19.1.1-373.g4e45 110 kB/s | 16 kB 00:00 2024-09-22T03:34:59.816 INFO:teuthology.orchestra.run.smithi117.stdout:(2/11): ceph-grafana-dashboards-19.1.1-373.g4e4 182 kB/s | 32 kB 00:00 2024-09-22T03:35:00.000 INFO:teuthology.orchestra.run.smithi117.stdout:(3/11): ceph-mgr-dashboard-19.1.1-373.g4e45e6b4 10 MB/s | 3.6 MB 00:00 2024-09-22T03:35:00.141 INFO:teuthology.orchestra.run.smithi117.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.8 MB/s | 548 kB 00:00 2024-09-22T03:35:00.167 INFO:teuthology.orchestra.run.smithi117.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 775 kB/s | 19 kB 00:00 2024-09-22T03:35:00.634 INFO:teuthology.orchestra.run.smithi117.stdout:(6/11): python3-protobuf-3.14.0-13.el9.noarch.r 320 kB/s | 269 kB 00:00 2024-09-22T03:35:00.726 INFO:teuthology.orchestra.run.smithi117.stdout:(7/11): protobuf-compiler-3.14.0-13.el9.x86_64. 949 kB/s | 863 kB 00:00 2024-09-22T03:35:00.751 INFO:teuthology.orchestra.run.smithi117.stdout:(8/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-09-22T03:35:00.802 INFO:teuthology.orchestra.run.smithi117.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 409 kB/s | 31 kB 00:00 2024-09-22T03:35:00.843 INFO:teuthology.orchestra.run.smithi117.stdout:(10/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.0 MB/s | 2.0 MB 00:00 2024-09-22T03:35:00.869 INFO:teuthology.orchestra.run.smithi117.stdout:(11/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-09-22T03:35:00.874 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:00.874 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.9 MB/s | 7.7 MB 00:01 2024-09-22T03:35:00.908 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:00.927 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:00.927 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:01.144 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:01.144 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:01.625 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:01.716 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-09-22T03:35:01.755 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-09-22T03:35:01.861 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-09-22T03:35:01.990 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-09-22T03:35:02.042 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-09-22T03:35:02.126 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-09-22T03:35:02.169 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-09-22T03:35:02.206 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:35:02.237 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 9/11 2024-09-22T03:35:03.302 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 10/11 2024-09-22T03:35:03.338 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 11/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9 1/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noar 2/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9. 3/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-09-22T03:35:04.425 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-09-22T03:35:04.742 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-09-22T03:35:04.742 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:04.742 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:04.742 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-22T03:35:04.742 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:04.743 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:04.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-22T03:35:05.563 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:05 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:05.697 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 7.4 M 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-22T03:35:05.698 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-09-22T03:35:05.699 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:05.700 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 38 M 2024-09-22T03:35:05.700 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 212 M 2024-09-22T03:35:05.700 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:06.392 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-37 16 MB/s | 7.4 MB 00:00 2024-09-22T03:35:06.509 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-09-22T03:35:06.793 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 220 kB/s | 191 kB 00:00 2024-09-22T03:35:06.876 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-09-22T03:35:06.901 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 168 kB/s | 18 kB 00:00 2024-09-22T03:35:06.927 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 765 kB/s | 38 kB 00:00 2024-09-22T03:35:06.994 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): libgfortran-11.5.0-2.el9.x86_64.rpm 748 kB/s | 801 kB 00:01 2024-09-22T03:35:07.019 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-09-22T03:35:07.077 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-22T03:35:07.453 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 9.0 MB/s | 4.9 MB 00:00 2024-09-22T03:35:07.578 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.7 MB/s | 5.1 MB 00:00 2024-09-22T03:35:07.920 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-09-22T03:35:07.920 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:07.921 INFO:teuthology.orchestra.run.smithi117.stdout:Total 17 MB/s | 38 MB 00:02 2024-09-22T03:35:08.275 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:08.302 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:08.302 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:08.733 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:08.733 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:09.497 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:09.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-22T03:35:09.626 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:35:09.725 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:35:09.793 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-22T03:35:09.980 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-22T03:35:10.017 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-22T03:35:10.094 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-22T03:35:10.781 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-22T03:35:10.906 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-22T03:35:12.179 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:35:13.217 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-22T03:35:13.242 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 12/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6 1/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-22T03:35:15.784 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-22T03:35:16.150 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-22T03:35:16.150 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-22T03:35:16.151 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-22T03:35:16.152 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:16.152 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:16.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-09-22T03:35:17.131 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:17 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:17.264 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 49 k 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-22T03:35:17.265 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-09-22T03:35:17.266 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:17.267 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.2 M 2024-09-22T03:35:17.267 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-09-22T03:35:17.267 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:17.568 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 232 kB/s | 19 kB 00:00 2024-09-22T03:35:17.593 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 237 kB/s | 26 kB 00:00 2024-09-22T03:35:17.652 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): ceph-mgr-rook-19.1.1-373.g4e45e6b4.el9. 296 kB/s | 49 kB 00:00 2024-09-22T03:35:17.702 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-09-22T03:35:17.727 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-09-22T03:35:17.769 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 772 kB/s | 32 kB 00:00 2024-09-22T03:35:17.794 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 563 kB/s | 14 kB 00:00 2024-09-22T03:35:17.820 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 457 kB/s | 54 kB 00:00 2024-09-22T03:35:17.845 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.3 MB/s | 220 kB 00:00 2024-09-22T03:35:17.879 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-22T03:35:17.904 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-22T03:35:17.962 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-09-22T03:35:18.004 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.5 MB/s | 1.0 MB 00:00 2024-09-22T03:35:18.007 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:18.007 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-22T03:35:18.070 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:18.086 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:18.086 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:18.174 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:18.176 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:18.677 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:18.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-22T03:35:18.906 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-22T03:35:18.969 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-22T03:35:19.006 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-22T03:35:19.048 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-22T03:35:19.159 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-22T03:35:19.260 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-22T03:35:19.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-22T03:35:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-22T03:35:20.088 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-22T03:35:20.122 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-22T03:35:20.170 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-22T03:35:20.192 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:35:20.982 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 13/13 2024-09-22T03:35:20.982 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 1/13 2024-09-22T03:35:20.982 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-22T03:35:20.982 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-22T03:35:20.983 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-22T03:35:21.317 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:21.318 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:21.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-09-22T03:35:22.172 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:22 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:22.304 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 146 k 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:35:22.305 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-09-22T03:35:22.306 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:22.307 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 7.0 M 2024-09-22T03:35:22.307 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-09-22T03:35:22.307 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:22.572 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-09-22T03:35:22.598 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): ceph-mgr-cephadm-19.1.1-373.g4e45e6b4.el 705 kB/s | 146 kB 00:00 2024-09-22T03:35:22.656 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 990 kB/s | 58 kB 00:00 2024-09-22T03:35:22.690 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-09-22T03:35:22.715 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.2 MB/s | 74 kB 00:00 2024-09-22T03:35:22.799 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-22T03:35:22.801 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:22.802 INFO:teuthology.orchestra.run.smithi117.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-22T03:35:22.872 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:22.883 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:22.884 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:22.983 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:22.983 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:23.289 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:23.471 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-22T03:35:23.525 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-22T03:35:24.050 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-22T03:35:24.140 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-22T03:35:24.210 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-22T03:35:24.246 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 6/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 1/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-22T03:35:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-22T03:35:25.346 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:25.347 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:25.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-09-22T03:35:26.171 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:26 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:26.304 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:26.304 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:26.304 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:35:26.304 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 868 k 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-09-22T03:35:26.305 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 948 k 2024-09-22T03:35:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 3.1 M 2024-09-22T03:35:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:26.484 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 508 kB/s | 80 kB 00:00 2024-09-22T03:35:26.593 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.1.1-373.g4e45e6b4.el9.x86_6 3.2 MB/s | 868 kB 00:00 2024-09-22T03:35:26.593 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:26.593 INFO:teuthology.orchestra.run.smithi117.stdout:Total 3.2 MB/s | 948 kB 00:00 2024-09-22T03:35:26.605 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:26.626 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:26.626 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:26.707 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:26.708 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:26.901 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:27.055 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-22T03:35:27.117 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:35:27.769 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/2 2024-09-22T03:35:27.769 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/2 2024-09-22T03:35:28.064 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-22T03:35:28.064 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:28.064 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:28.064 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-22T03:35:28.065 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:28.065 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:28.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-09-22T03:35:28.947 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:28 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:29.081 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:29.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:29.082 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:35:29.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.1.1-373.g4e45e6b4.el9 ceph-noarch 285 k 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 285 k 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.3 M 2024-09-22T03:35:29.083 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-volume-19.1.1-373.g4e45e6b4.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-22T03:35:29.314 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-22T03:35:29.315 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:29.323 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:29.350 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:29.351 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:29.565 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:29.633 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:35:29.649 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:35:29.649 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:35:29.649 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-22T03:35:29.649 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 1/1 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.1.1-373.g4e45e6b4.el9.noarch 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:30.671 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:30.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-09-22T03:35:31.559 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:31 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:31.694 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 127 k 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:31.695 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-09-22T03:35:31.696 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 457 k 2024-09-22T03:35:31.696 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:31.886 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.1.1-373.g4e45e6b4.el9.x86_64. 669 kB/s | 127 kB 00:00 2024-09-22T03:35:31.886 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:31.887 INFO:teuthology.orchestra.run.smithi117.stdout:Total 663 kB/s | 127 kB 00:00 2024-09-22T03:35:31.887 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:31.892 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:31.892 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:31.956 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:31.956 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:32.109 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:32.218 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:32.653 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:32.932 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:32.932 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:32.932 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:32.933 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:35:32.933 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:32.933 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:33.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-09-22T03:35:33.782 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:33 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:33.872 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:33.916 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:33.918 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:33.918 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:33.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-09-22T03:35:34.573 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:34 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:34.705 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 32 k 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:34.706 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:34.707 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 32 k 2024-09-22T03:35:34.707 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 142 k 2024-09-22T03:35:34.707 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:34.856 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.1.1-373.g4e45e6b4.el9.x86_64 213 kB/s | 32 kB 00:00 2024-09-22T03:35:34.857 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:34.857 INFO:teuthology.orchestra.run.smithi117.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-22T03:35:34.857 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:34.860 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:34.861 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:34.880 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:34.880 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:34.970 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:35.103 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:35.485 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:35.729 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:35.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-09-22T03:35:36.500 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:36 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:36.590 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:36.633 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:36.634 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:36.634 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:36.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-09-22T03:35:37.301 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:37 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:37.390 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:37.434 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:37.435 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:37.435 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:37.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-09-22T03:35:38.096 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:38 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:38.185 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:38.229 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:38.230 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:38.230 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:38.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-09-22T03:35:38.875 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:38 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:38.964 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:39.008 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:39.008 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:39.009 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:39.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-09-22T03:35:39.661 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:39 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:39.750 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:39.792 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:39.793 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:39.794 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:39.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-09-22T03:35:40.446 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:40 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:40.536 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 is already installed. 2024-09-22T03:35:40.579 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:40.580 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-09-22T03:35:40.580 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:40.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-09-22T03:35:41.237 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:41 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:41.371 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 85 k 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:41.372 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:41.373 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-09-22T03:35:41.373 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 256 k 2024-09-22T03:35:41.373 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:41.558 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 461 kB/s | 85 kB 00:00 2024-09-22T03:35:41.559 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:41.559 INFO:teuthology.orchestra.run.smithi117.stdout:Total 458 kB/s | 85 kB 00:00 2024-09-22T03:35:41.559 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:41.566 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:41.566 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:41.630 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:41.630 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:41.775 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:41.880 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:42.266 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:42.528 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:42.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-09-22T03:35:43.414 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:43 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 3.1 M 2024-09-22T03:35:43.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 14 M 2024-09-22T03:35:43.562 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:43.879 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 9.6 MB/s | 3.1 MB 00:00 2024-09-22T03:35:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:43.881 INFO:teuthology.orchestra.run.smithi117.stdout:Total 9.6 MB/s | 3.1 MB 00:00 2024-09-22T03:35:43.881 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:43.896 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:43.897 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:44.020 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:44.020 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:44.555 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-22T03:35:44.582 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:45.689 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:45.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-09-22T03:35:46.561 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:02:46 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:35:46.694 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:35:46.695 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 169 k 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:46.696 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 169 k 2024-09-22T03:35:46.697 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 540 k 2024-09-22T03:35:46.697 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:35:46.908 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.1.1-373.g4e45e6b4.el9.x86_64.rpm 802 kB/s | 169 kB 00:00 2024-09-22T03:35:46.908 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:35:46.908 INFO:teuthology.orchestra.run.smithi117.stdout:Total 796 kB/s | 169 kB 00:00 2024-09-22T03:35:46.909 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:35:46.916 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:35:46.916 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:35:46.981 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:35:46.981 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:35:47.131 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:35:47.222 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:47.674 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:47.941 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/1 2024-09-22T03:35:47.942 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:47.942 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:35:47.942 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:35:47.942 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:47.942 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:35:48.205 DEBUG:teuthology.parallel:result is None 2024-09-22T03:35:48.206 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-22T03:35:48.347 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-22T03:35:48.369 INFO:teuthology.orchestra.run.smithi117.stdout:19.1.1-373.g4e45e6b4.el9 2024-09-22T03:35:48.369 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373.g4e45e6b4.el9 2024-09-22T03:35:48.369 INFO:teuthology.task.install:The correct ceph version 19.1.1-373.g4e45e6b4 is installed. 2024-09-22T03:35:48.371 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-22T03:35:48.371 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:48.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-22T03:35:48.434 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-22T03:35:48.434 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:48.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-09-22T03:35:48.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-22T03:35:48.563 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-22T03:35:48.563 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:48.563 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-22T03:35:48.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-22T03:35:48.691 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-22T03:35:48.693 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:48.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-09-22T03:35:48.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-22T03:35:48.819 INFO:teuthology.run_tasks:Running task ceph... 2024-09-22T03:35:48.899 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-22T03:35:48.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-09-22T03:35:48.925 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-22T03:35:48.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-22T03:35:48.990 INFO:tasks.ceph:Creating extra log directories... 2024-09-22T03:35:48.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-22T03:35:49.062 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-22T03:35:49.062 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-22T03:35:49.062 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785', 'branch': 'squid', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914785', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1158, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '0/1', 'suite': 'crimson-rados', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPS2s2o+lsmnwUlZ00b5vyZQeHztirg0JS2Z8OGAUPRSkb5Yo9neStTCLsmdnDsbziSrD/COEboD8/j+AlEwl7E='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:56:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-22T03:35:49.063 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-22T03:35:49.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-09-22T03:35:49.174 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:49.175 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-09-22T03:35:49.230 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-22T03:35:49.230 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-22 03:35:44.838174076 +0000 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-22 03:32:21.258152948 +0000 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-22 03:32:21.258152948 +0000 2024-09-22T03:35:49.285 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-22 03:32:21.258152948 +0000 2024-09-22T03:35:49.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-22T03:35:49.348 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-22T03:35:49.348 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-22T03:35:49.348 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.00020434 s, 2.5 MB/s 2024-09-22T03:35:49.349 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-22T03:35:49.405 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-09-22T03:35:49.461 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-22T03:35:49.461 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:35:49.461 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-22 03:35:44.839174057 +0000 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-22 03:32:21.595146382 +0000 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-22 03:32:21.595146382 +0000 2024-09-22T03:35:49.462 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-22 03:32:21.595146382 +0000 2024-09-22T03:35:49.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-22T03:35:49.524 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-22T03:35:49.524 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-22T03:35:49.524 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000181284 s, 2.8 MB/s 2024-09-22T03:35:49.525 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-22T03:35:49.581 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-09-22T03:35:49.636 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-22T03:35:49.636 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:35:49.636 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-22 03:35:44.839174057 +0000 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-22 03:32:21.909140262 +0000 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-22 03:32:21.909140262 +0000 2024-09-22T03:35:49.637 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-22 03:32:21.909140262 +0000 2024-09-22T03:35:49.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-22T03:35:49.699 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-22T03:35:49.699 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-22T03:35:49.699 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000200355 s, 2.6 MB/s 2024-09-22T03:35:49.700 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-22T03:35:49.756 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-09-22T03:35:49.812 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-09-22 03:35:44.839174057 +0000 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-09-22 03:32:22.236133891 +0000 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-09-22 03:32:22.236133891 +0000 2024-09-22T03:35:49.813 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-09-22 03:32:22.236133891 +0000 2024-09-22T03:35:49.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-22T03:35:49.876 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-09-22T03:35:49.876 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-09-22T03:35:49.877 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000169371 s, 3.0 MB/s 2024-09-22T03:35:49.878 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-22T03:35:49.935 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:35:49.935 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:35:49.935 INFO:tasks.ceph:Generating config... 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] osd objectstore = seastore 2024-09-22T03:35:49.936 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] auth client required = none 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] auth cluster required = none 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] auth service required = none 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] auth supported = none 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug asok = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug auth = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug client = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug context = 0/0 2024-09-22T03:35:49.937 INFO:tasks.ceph:[global] debug crush = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug filer = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug journal = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-09-22T03:35:49.938 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug mon = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug monc = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug ms = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-09-22T03:35:49.939 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug osd = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rados = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug refs = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug striper = 0/0 2024-09-22T03:35:49.940 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] debug timer = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] debug tp = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms client mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[global] ms service mode = crc 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-09-22T03:35:49.941 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-09-22T03:35:49.942 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-09-22T03:35:49.942 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-09-22T03:35:49.942 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-09-22T03:35:49.942 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-09-22T03:35:49.942 INFO:tasks.ceph:Setting up mon.a... 2024-09-22T03:35:49.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-22T03:35:50.032 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.keyring 2024-09-22T03:35:50.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-22T03:35:50.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:35:50.107 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.117')] 2024-09-22T03:35:50.107 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.117', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'osd objectstore': 'seastore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-09-22T03:35:50.108 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:50.108 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:35:50.162 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.117 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:50.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:35:50.247+0000 7f4abda33e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:50.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:35:50.247+0000 7f4abda33e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:50.248 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:35:50.247+0000 7f4abda33e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: generated fsid 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:setting min_mon_release = quincy 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:epoch 0 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:fsid 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:last_changed 2024-09-22T03:35:50.248450+0000 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:created 2024-09-22T03:35:50.248450+0000 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:min_mon_release 17 (quincy) 2024-09-22T03:35:50.249 INFO:teuthology.orchestra.run.smithi117.stdout:election_strategy: 1 2024-09-22T03:35:50.250 INFO:teuthology.orchestra.run.smithi117.stdout:0: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.a 2024-09-22T03:35:50.250 INFO:teuthology.orchestra.run.smithi117.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-09-22T03:35:50.251 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-22T03:35:50.265 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7281d05b-e4e0-4ef1-9db0-6fb22de06150... 2024-09-22T03:35:50.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = none 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-09-22T03:35:50.347 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.348 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.349 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = 172.21.15.117 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: auth client required = none 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: auth cluster required = none 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: auth service required = none 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug asok = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug bdev = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug buffer = 0/0 2024-09-22T03:35:50.350 INFO:teuthology.orchestra.run.smithi117.stdout: debug civetweb = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug client = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug compressor = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug context = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug crush = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug crypto = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug dpdk = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug eventtrace = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug filer = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug filestore = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug finisher = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug fuse = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug heartbeatmap = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug javaclient = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug journal = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug journaler = 0/0 2024-09-22T03:35:50.351 INFO:teuthology.orchestra.run.smithi117.stdout: debug kstore = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug leveldb = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug lockdep = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds balancer = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds locker = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds log = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds log expire = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds migrator = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug memdb = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgrc = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 0/0 2024-09-22T03:35:50.352 INFO:teuthology.orchestra.run.smithi117.stdout: debug objclass = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug objectcacher = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug optracker = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug perfcounter = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd mirror = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rbd replay = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug refs = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rgw = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rgw lifecycle = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rgw sync = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 0/0 2024-09-22T03:35:50.353 INFO:teuthology.orchestra.run.smithi117.stdout: debug striper = 0/0 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: debug throttle = 0/0 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: debug timer = 0/0 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: debug tp = 0/0 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms client mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms cluster mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms mon client mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms mon cluster mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms mon service mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: ms service mode = crc 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-09-22T03:35:50.354 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.355 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: crimson alien thread cpu cores = 6-7 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: crimson osd obc lru size = 10 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: debug filestore = 0/0 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: debug journal = 0/0 2024-09-22T03:35:50.356 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 0/0 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 0/0 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = seastore 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-09-22T03:35:50.357 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 0/0 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 0/0 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 0/0 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-09-22T03:35:50.358 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default crimson = True 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-09-22T03:35:50.359 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: crimson seastar cpu cores = 0-2 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-09-22T03:35:50.360 INFO:teuthology.orchestra.run.smithi117.stdout: crimson seastar cpu cores = 3-5 2024-09-22T03:35:50.361 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-22T03:35:50.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-22T03:35:50.461 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-22T03:35:50.461 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:50.461 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-22T03:35:50.476 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:50.476 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-22T03:35:50.531 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:35:50.531 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:50.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-22T03:35:50.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-22T03:35:50.605 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:35:50.605 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:50.660 INFO:tasks.ceph:Setting up mon nodes... 2024-09-22T03:35:50.660 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-22T03:35:50.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:35:50.756 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-22T03:35:50.759 INFO:tasks.ceph:Setting up mds nodes... 2024-09-22T03:35:50.759 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-22T03:35:50.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-22T03:35:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-22T03:35:50.818 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-22T03:35:50.818 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-22T03:35:50.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-22T03:35:50.843 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:35:50.843 INFO:tasks.ceph:role: osd.0 2024-09-22T03:35:50.844 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:35:50.844 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:35:50.916 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:35:50.917 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:35:50.917 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:35:51.429 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-22T03:35:51.431 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-22T03:35:51.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-22T03:35:51.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-22T03:35:51.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-22T03:35:51.646 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:35:51.646 INFO:tasks.ceph:role: osd.1 2024-09-22T03:35:51.646 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:35:51.646 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:35:51.719 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:35:52.182 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-22T03:35:52.184 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-22T03:35:52.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-22T03:35:52.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-22T03:35:52.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-22T03:35:52.402 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-22T03:35:52.402 INFO:tasks.ceph:role: osd.2 2024-09-22T03:35:52.403 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:35:52.403 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-22T03:35:52.474 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-22T03:35:52.475 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-22T03:35:52.984 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-09-22T03:35:52.986 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-09-22T03:35:52.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-22T03:35:53.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-22T03:35:53.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:53.689 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:53.690 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:53,690 seastar - Reactor backend: linux-aio 2024-09-22T03:35:53.707 INFO:teuthology.orchestra.run.smithi117.stderr:==28094==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:53.762 INFO:teuthology.orchestra.run.smithi117.stderr:DEBUG 2024-09-22 03:35:53,762 [shard 0:main] seastar - reactor::drain 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:54.253 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:35:54.311 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:54.312 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:54,312 seastar - Reactor backend: linux-aio 2024-09-22T03:35:54.348 INFO:teuthology.orchestra.run.smithi117.stderr:==28092==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:54.497 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c605 0x2857c 0xb6323 0x2ffbca31 0x3014d7dd 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:54.497 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff99e19342 0xffffffff99e20b38 0xffffffff99d8e01c 0xffffffff99d9699b 0xffffffff99d96e1c 0xffffffff99a8c884 0xffffffff9a6d1db2 0xffffffff9a800bb2 2024-09-22T03:35:54.497 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c5f1 0x282da 0xb49e1 0x2bd7ed84 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:54.498 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff99d8eae0 0xffffffff99d9699b 0xffffffff99d96e1c 0xffffffff99a8c884 0xffffffff9a6d1db2 0xffffffff9a800bb2 2024-09-22T03:35:54.792 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x30144b9a 0x30145a3b 0x3014b444 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:54.793 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:54.793 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xba73f 0xc384c 0x2c57d 0x2857c 0xb6323 0x3003e4de 0x3003ed8a 0x30041e94 0x3004afd5 0x3014aa5f 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:54.793 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:55.526 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:35:55.527 INFO:teuthology.orchestra.run.smithi117.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:35:55.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-22T03:35:55.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:56.141 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:56.142 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:56,142 seastar - Reactor backend: linux-aio 2024-09-22T03:35:56.160 INFO:teuthology.orchestra.run.smithi117.stderr:==28141==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:56.206 INFO:teuthology.orchestra.run.smithi117.stderr:DEBUG 2024-09-22 03:35:56,206 [shard 0:main] seastar - reactor::drain 2024-09-22T03:35:56.700 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:56.701 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:35:56.701 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:35:56.701 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:56.701 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:56.701 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:35:56.757 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:56.758 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:56,758 seastar - Reactor backend: linux-aio 2024-09-22T03:35:56.786 INFO:teuthology.orchestra.run.smithi117.stderr:==28139==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:56.931 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd59f3 0xd554b 0x2c64a 0x2857c 0xb6323 0x2fcc7347 0x3015162d 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:56.931 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:56.931 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff762f6 0x2ff7730f 0x3014bdcc 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:56.931 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:57.247 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2fcc61dd 0x30150a86 0x2ff3c411 0x2fd3eb74 0x2bd7c6de 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:57.247 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:57.247 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x28790 0xb738c 0x24eabcb7 0x24eabd21 0x3014abaf 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:57.247 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:57.304 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 65 ms on shard 0, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095a19 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047733a 0x2fdaa7b9 0x2fdac545 0x208bac63 0x2958f 0x2963f 0x20471d84 2024-09-22T03:35:57.304 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:57.337 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 65 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2c095963 0x2c21e1fb 0x2c132403 0x2ba0ddbf 0x2ba105d4 0x2ba10f8b 0x2ba11587 0x2bb947dc 0x2b50cf98 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:57.337 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:57.973 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:57.973 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:35:57.973 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:35:57.974 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:57.974 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:57.974 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:35:57.974 INFO:teuthology.orchestra.run.smithi117.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:35:58.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-22T03:35:58.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:35:58.588 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:58.589 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:58,588 seastar - Reactor backend: linux-aio 2024-09-22T03:35:58.606 INFO:teuthology.orchestra.run.smithi117.stderr:==28188==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:58.662 INFO:teuthology.orchestra.run.smithi117.stderr:DEBUG 2024-09-22 03:35:58,661 [shard 0:main] seastar - reactor::drain 2024-09-22T03:35:59.155 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:59.155 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:35:59.155 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:35:59.155 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:35:59.156 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:35:59.156 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:35:59.210 INFO:teuthology.orchestra.run.smithi117.stderr:WARNING: debug mode. Not for benchmarking or production 2024-09-22T03:35:59.211 INFO:teuthology.orchestra.run.smithi117.stderr:INFO 2024-09-22 03:35:59,211 seastar - Reactor backend: linux-aio 2024-09-22T03:35:59.247 INFO:teuthology.orchestra.run.smithi117.stderr:==28186==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-09-22T03:35:59.396 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0xd5d9f 0xc0fec 0xb73f3 0x24eabcb7 0x24eabd21 0x2abc286b 0x2fd9a4c6 0x2bd80b68 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:59.396 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:59.396 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x8b56c 0xc384c 0x2c57d 0x2857c 0xb6323 0x2fcc5109 0x2fcc5a6e 0x3015162d 0x2ff3c411 0x2fd3eb74 0x2bd80a7f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2acacc1c 0x2b242299 0x2b3b1886 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:59.396 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:59.696 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x2ff765ba 0x2ff7730f 0x3014bdcc 0x30153185 0x2ff3c411 0x2fd3eb74 0x2bd83d03 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:59.697 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:35:59.697 INFO:teuthology.orchestra.run.smithi117.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: 0x45d5d 0x302dc944 0x300f44c5 0x3021b180 0x3021b504 0x3021b64e 0x3021baa4 0x3e6ef 0x39753 0x94456 0x94d46 0x205d41f0 0x300714e0 0x30071f09 0x3007230a 0x3007d9e0 0x30081d6a 0x2b8ab682 0x2bd83c3f 0x2bd97077 0x2bb2a603 0x2b2859fc 0x2aca5660 0x2b24a027 0x2b39eea2 0x2b60ab92 0x30238775 0x302f6bae 0x304754e4 0x3047b292 0x3047b814 0x30fab151 0x89c51 0x10ec7f 2024-09-22T03:35:59.697 INFO:teuthology.orchestra.run.smithi117.stderr:kernel callstack: 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr:Suppressions used: 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr: count bytes template 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr: 1 8 MallocExtension::Register 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr:----------------------------------------------------- 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:36:00.439 INFO:teuthology.orchestra.run.smithi117.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 7281d05b-e4e0-4ef1-9db0-6fb22de06150 2024-09-22T03:36:00.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-22T03:36:00.511 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-22T03:36:00.511 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:00.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-22T03:36:00.575 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:00.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-22T03:36:00.639 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:00.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-22T03:36:00.703 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:00.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-22T03:36:00.768 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:00.768 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-22T03:36:00.823 INFO:tasks.ceph:Adding keys to all mons... 2024-09-22T03:36:00.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCWkO9mTPgQLRAALBbJA3la8P4c0Qdh4LXX7w== 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCakO9maRwGGBAAK5/2k4UbdK4zwlcbkKSYBg== 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCckO9moFARMhAAzvNzYKXhkU9cWxyOt9c0xg== 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCfkO9m9fQCEhAA7NmVDU5tKOT/J4Ipelwtdg== 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-09-22T03:36:00.886 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQCWkO9mmcL+LxAAGEtRJ+ghDtLympUVhTJvyw== 2024-09-22T03:36:00.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-22T03:36:00.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:36:01.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:36:01.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-22T03:36:01.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-22T03:36:01.209 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-22T03:36:01.209 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-22T03:36:01.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-22T03:36:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.333+0000 7f6c418a4b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.333+0000 7f6c418a4b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-22T03:36:01.371 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-22T03:36:01.426 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-22T03:36:01.426 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-22T03:36:01.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-22T03:36:01.468 INFO:tasks.ceph.mon.a:Started 2024-09-22T03:36:01.468 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-22T03:36:01.469 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-22T03:36:01.469 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-22T03:36:01.471 INFO:tasks.ceph.mgr.x:Started 2024-09-22T03:36:01.471 INFO:tasks.ceph:Setting crush tunables to default 2024-09-22T03:36:01.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd crush tunables default 2024-09-22T03:36:01.546 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-09-22T03:36:01.544+0000 7fb5430b1b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.546 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-09-22T03:36:01.544+0000 7fb5430b1b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.546 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-09-22T03:36:01.544+0000 7fb5430b1b40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.547 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:01.546+0000 7fec87a88f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.570+0000 7f6e794c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.577+0000 7f6e794c3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.644 INFO:teuthology.orchestra.run.smithi117.stderr:adjusted tunables profile to default 2024-09-22T03:36:01.656 INFO:tasks.ceph:check_enable_crimson: True 2024-09-22T03:36:01.656 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-09-22T03:36:01.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-09-22T03:36:01.751 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:01.750+0000 7fec87a88f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.751 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:01.750+0000 7fec87a88f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.764 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.758+0000 7f9cd7b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.766 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:01.765+0000 7f9cd7b17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:01.872 INFO:tasks.ceph.mon.a.smithi117.stderr:2024-09-22T03:36:01.871+0000 7fb53aff1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-09-22T03:36:01.872 INFO:tasks.ceph.mon.a.smithi117.stderr:Please be aware that this feature is experimental, untested, 2024-09-22T03:36:01.872 INFO:tasks.ceph.mon.a.smithi117.stderr:unsupported, and may result in data corruption, data loss, 2024-09-22T03:36:01.872 INFO:tasks.ceph.mon.a.smithi117.stderr:and/or irreparable damage to your cluster. Do not use 2024-09-22T03:36:01.873 INFO:tasks.ceph.mon.a.smithi117.stderr:feature with important data. 2024-09-22T03:36:01.873 INFO:tasks.ceph.mon.a.smithi117.stderr: 2024-09-22T03:36:01.873 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:01.872+0000 7fec87a88f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-22T03:36:01.913 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:01.912+0000 7fec87a88f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-22T03:36:02.054 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.052+0000 7fec87a88f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-22T03:36:02.103 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.102+0000 7fec87a88f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-22T03:36:02.269 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.268+0000 7fec87a88f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:36:02.377 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.375+0000 7fec87a88f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-22T03:36:02.656 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-22T03:36:02.656 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:02.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-22T03:36:02.681 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:02.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-22T03:36:02.694 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.693+0000 7fec87a88f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-22T03:36:02.719 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:02.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-22T03:36:02.760 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.758+0000 7fec87a88f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-22T03:36:02.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 309aae80-60cd-4edb-b59a-9ed18c644226 0 2024-09-22T03:36:02.917 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:02.916+0000 7fec87a88f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-22T03:36:02.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:02.920+0000 7f5ea9757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:02.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:02.927+0000 7f5ea9757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.001 INFO:teuthology.orchestra.run.smithi117.stdout:0 2024-09-22T03:36:03.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new ae70916b-8f50-47cf-bf8e-d99448280b2e 1 2024-09-22T03:36:03.115 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.108+0000 7fc7fc22c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.117 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.116+0000 7fc7fc22c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.190 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-09-22T03:36:03.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new e7b807d2-ff30-44a5-b60d-c907b19a9199 2 2024-09-22T03:36:03.306 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.299+0000 7f8d802ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.308 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.306+0000 7f8d802ce640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.420 INFO:teuthology.orchestra.run.smithi117.stdout:2 2024-09-22T03:36:03.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-09-22T03:36:03.462 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-22T03:36:03.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-09-22T03:36:03.506 INFO:tasks.ceph.osd.0:Started 2024-09-22T03:36:03.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-09-22T03:36:03.536 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-22T03:36:03.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-09-22T03:36:03.579 INFO:tasks.ceph.osd.1:Started 2024-09-22T03:36:03.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-09-22T03:36:03.610 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-22T03:36:03.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-09-22T03:36:03.653 INFO:tasks.ceph.osd.2:Started 2024-09-22T03:36:03.653 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:36:03.839 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.830+0000 7f9f9c797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.842 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:03.841+0000 7f9f9c797640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:03.973 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:03.974 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":6,"fsid":"7281d05b-e4e0-4ef1-9db0-6fb22de06150","created":"2024-09-22T03:36:01.563009+0000","modified":"2024-09-22T03:36:03.419382+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:36:03.419382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"309aae80-60cd-4edb-b59a-9ed18c644226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae70916b-8f50-47cf-bf8e-d99448280b2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7b807d2-ff30-44a5-b60d-c907b19a9199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:36:03.996 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-22T03:36:03.996 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-22T03:36:04.123 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.122+0000 7fec87a88f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-22T03:36:04.210 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.209+0000 7fec87a88f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-22T03:36:04.299 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:36:04.397 INFO:teuthology.misc.health.smithi117.stderr:2024-09-22T03:36:04.392+0000 7f4e8050e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:04.399 INFO:teuthology.misc.health.smithi117.stderr:2024-09-22T03:36:04.398+0000 7f4e8050e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:04.464 INFO:teuthology.misc.health.smithi117.stdout: 2024-09-22T03:36:04.464 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":6,"fsid":"7281d05b-e4e0-4ef1-9db0-6fb22de06150","created":"2024-09-22T03:36:01.563009+0000","modified":"2024-09-22T03:36:03.419382+0000","last_up_change":"0.000000","last_in_change":"2024-09-22T03:36:03.419382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"309aae80-60cd-4edb-b59a-9ed18c644226","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae70916b-8f50-47cf-bf8e-d99448280b2e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7b807d2-ff30-44a5-b60d-c907b19a9199","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:36:04.475 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-22T03:36:04.627 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.626+0000 7fec87a88f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-22T03:36:04.721 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.720+0000 7fec87a88f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-22T03:36:04.837 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.835+0000 7fec87a88f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-22T03:36:04.887 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:04.886+0000 7fec87a88f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-22T03:36:05.010 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.008+0000 7fec87a88f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-22T03:36:05.067 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.066+0000 7fec87a88f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-22T03:36:05.117 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.115+0000 7fec87a88f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-22T03:36:05.235 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.234+0000 7fec87a88f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-22T03:36:05.288 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.286+0000 7fec87a88f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-22T03:36:05.341 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.340+0000 7fec87a88f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-22T03:36:05.411 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.410+0000 7fec87a88f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-22T03:36:05.458 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.457+0000 7fec87a88f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-22T03:36:05.504 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:05.502+0000 7fec87a88f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-22T03:36:06.264 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:06.262+0000 7fec87a88f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-22T03:36:07.274 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-09-22T03:36:07.273+0000 7fec26c2d640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-22T03:36:10.778 DEBUG:teuthology.orchestra.run.smithi117:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-22T03:36:10.886 INFO:teuthology.misc.health.smithi117.stderr:2024-09-22T03:36:10.882+0000 7ff5df517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:10.888 INFO:teuthology.misc.health.smithi117.stderr:2024-09-22T03:36:10.887+0000 7ff5df517640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:11.064 INFO:teuthology.misc.health.smithi117.stdout: 2024-09-22T03:36:11.064 INFO:teuthology.misc.health.smithi117.stdout:{"epoch":10,"fsid":"7281d05b-e4e0-4ef1-9db0-6fb22de06150","created":"2024-09-22T03:36:01.563009+0000","modified":"2024-09-22T03:36:10.327972+0000","last_up_change":"2024-09-22T03:36:08.322088+0000","last_in_change":"2024-09-22T03:36:03.419382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:36:10.309479+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"309aae80-60cd-4edb-b59a-9ed18c644226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6807","nonce":1630114081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1630114081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6809","nonce":1630114081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1630114081}]},"public_addr":"172.21.15.117:6807/1630114081","cluster_addr":"172.21.15.117:6806/1630114081","heartbeat_back_addr":"172.21.15.117:6809/1630114081","heartbeat_front_addr":"172.21.15.117:6808/1630114081","state":["exists","up"]},{"osd":1,"uuid":"ae70916b-8f50-47cf-bf8e-d99448280b2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2497309198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6801","nonce":2497309198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6805","nonce":2497309198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2497309198}]},"public_addr":"172.21.15.117:6800/2497309198","cluster_addr":"172.21.15.117:6801/2497309198","heartbeat_back_addr":"172.21.15.117:6805/2497309198","heartbeat_front_addr":"172.21.15.117:6804/2497309198","state":["exists","up"]},{"osd":2,"uuid":"e7b807d2-ff30-44a5-b60d-c907b19a9199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1195809028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1195809028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1195809028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6811","nonce":1195809028}]},"public_addr":"172.21.15.117:6813/1195809028","cluster_addr":"172.21.15.117:6812/1195809028","heartbeat_back_addr":"172.21.15.117:6810/1195809028","heartbeat_front_addr":"172.21.15.117:6811/1195809028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:36:11.075 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-22T03:36:11.075 INFO:tasks.ceph:Creating RBD pool 2024-09-22T03:36:11.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-22T03:36:11.178 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:11.171+0000 7fe255f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:11.180 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:11.179+0000 7fe255f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:12.337 INFO:teuthology.orchestra.run.smithi117.stderr:pool 'rbd' created 2024-09-22T03:36:12.353 DEBUG:teuthology.orchestra.run.smithi117:> rbd --cluster ceph pool init rbd 2024-09-22T03:36:12.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:12.404+0000 7fe03542fd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:12.406 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:12.404+0000 7fe03542fd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:12.407 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:12.406+0000 7fe03542fd00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:15.396 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-22T03:36:15.397 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-22T03:36:15.397 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-22T03:36:15.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-22T03:36:15.398 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-22T03:36:15.505 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:15.501+0000 7ff24e26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:15.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:15.506+0000 7ff24e26c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:15.727 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:15.738 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1714674228},{"type":"v1","addr":"172.21.15.117:6803","nonce":1714674228}]},"active_addr":"172.21.15.117:6803/1714674228","active_change":"2024-09-22T03:36:06.265625+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":330696445}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2883443699}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":1846538316}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":9994764}]}]} 2024-09-22T03:36:15.742 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-22T03:36:15.742 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-22T03:36:15.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:36:15.844 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:15.840+0000 7fac36c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:15.846 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:15.845+0000 7fac36c2c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.021 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:16.021 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"fsid":"7281d05b-e4e0-4ef1-9db0-6fb22de06150","created":"2024-09-22T03:36:01.563009+0000","modified":"2024-09-22T03:36:15.350551+0000","last_up_change":"2024-09-22T03:36:08.322088+0000","last_in_change":"2024-09-22T03:36:03.419382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:36:10.309479+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:36:11.398826+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"309aae80-60cd-4edb-b59a-9ed18c644226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6807","nonce":1630114081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1630114081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6809","nonce":1630114081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1630114081}]},"public_addr":"172.21.15.117:6807/1630114081","cluster_addr":"172.21.15.117:6806/1630114081","heartbeat_back_addr":"172.21.15.117:6809/1630114081","heartbeat_front_addr":"172.21.15.117:6808/1630114081","state":["exists","up"]},{"osd":1,"uuid":"ae70916b-8f50-47cf-bf8e-d99448280b2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2497309198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6801","nonce":2497309198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6805","nonce":2497309198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2497309198}]},"public_addr":"172.21.15.117:6800/2497309198","cluster_addr":"172.21.15.117:6801/2497309198","heartbeat_back_addr":"172.21.15.117:6805/2497309198","heartbeat_front_addr":"172.21.15.117:6804/2497309198","state":["exists","up"]},{"osd":2,"uuid":"e7b807d2-ff30-44a5-b60d-c907b19a9199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1195809028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1195809028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1195809028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6811","nonce":1195809028}]},"public_addr":"172.21.15.117:6813/1195809028","cluster_addr":"172.21.15.117:6812/1195809028","heartbeat_back_addr":"172.21.15.117:6810/1195809028","heartbeat_front_addr":"172.21.15.117:6811/1195809028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:36:16.034 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-22T03:36:16.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-22T03:36:16.137 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.133+0000 7f2aa7157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.139 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.138+0000 7f2aa7157640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.318 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:16.318 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":15,"fsid":"7281d05b-e4e0-4ef1-9db0-6fb22de06150","created":"2024-09-22T03:36:01.563009+0000","modified":"2024-09-22T03:36:15.350551+0000","last_up_change":"2024-09-22T03:36:08.322088+0000","last_in_change":"2024-09-22T03:36:03.419382+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-22T03:36:10.309479+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-22T03:36:11.398826+0000","flags":270369,"flags_names":"hashpspool,nopgchange,selfmanaged_snaps,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"309aae80-60cd-4edb-b59a-9ed18c644226","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6807","nonce":1630114081}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1630114081}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6809","nonce":1630114081}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1630114081}]},"public_addr":"172.21.15.117:6807/1630114081","cluster_addr":"172.21.15.117:6806/1630114081","heartbeat_back_addr":"172.21.15.117:6809/1630114081","heartbeat_front_addr":"172.21.15.117:6808/1630114081","state":["exists","up"]},{"osd":1,"uuid":"ae70916b-8f50-47cf-bf8e-d99448280b2e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":2497309198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6801","nonce":2497309198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6805","nonce":2497309198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":2497309198}]},"public_addr":"172.21.15.117:6800/2497309198","cluster_addr":"172.21.15.117:6801/2497309198","heartbeat_back_addr":"172.21.15.117:6805/2497309198","heartbeat_front_addr":"172.21.15.117:6804/2497309198","state":["exists","up"]},{"osd":2,"uuid":"e7b807d2-ff30-44a5-b60d-c907b19a9199","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6813","nonce":1195809028}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1195809028}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1195809028}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6811","nonce":1195809028}]},"public_addr":"172.21.15.117:6813/1195809028","cluster_addr":"172.21.15.117:6812/1195809028","heartbeat_back_addr":"172.21.15.117:6810/1195809028","heartbeat_front_addr":"172.21.15.117:6811/1195809028","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-22T03:36:16.330 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-22T03:36:16.330 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-22T03:36:16.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-22T03:36:16.440 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.435+0000 7f2734757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.435+0000 7f1192218640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.441 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.435+0000 7f189f597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.442+0000 7f1192218640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.442+0000 7f189f597640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.442+0000 7f2734757640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.467 INFO:teuthology.orchestra.run.smithi117.stdout:34359738377 2024-09-22T03:36:16.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-22T03:36:16.470 INFO:teuthology.orchestra.run.smithi117.stdout:38654705672 2024-09-22T03:36:16.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-22T03:36:16.472 INFO:teuthology.orchestra.run.smithi117.stdout:38654705672 2024-09-22T03:36:16.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-22T03:36:16.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.569+0000 7f6145557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.575+0000 7f6145557640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.616+0000 7fe7c0f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.621 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.616+0000 7f2a4842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.622+0000 7f2a4842c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.623 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.622+0000 7fe7c0f97640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.763 INFO:teuthology.orchestra.run.smithi117.stdout:34359738377 2024-09-22T03:36:16.780 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738377 got 34359738377 for osd.1 2024-09-22T03:36:16.780 DEBUG:teuthology.parallel:result is None 2024-09-22T03:36:16.812 INFO:teuthology.orchestra.run.smithi117.stdout:38654705672 2024-09-22T03:36:16.813 INFO:teuthology.orchestra.run.smithi117.stdout:38654705672 2024-09-22T03:36:16.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.0 2024-09-22T03:36:16.826 DEBUG:teuthology.parallel:result is None 2024-09-22T03:36:16.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705672 got 38654705672 for osd.2 2024-09-22T03:36:16.826 DEBUG:teuthology.parallel:result is None 2024-09-22T03:36:16.827 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-22T03:36:16.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:36:16.969 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.964+0000 7ffba3b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:16.971 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:16.970+0000 7ffba3b57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:17.141 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:17.141 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-22T03:36:17.153 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T03:36:16.460300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3456,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889728,"statfs":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366810+0000","last_change":"2024-09-22T03:36:15.369055+0000","last_active":"2024-09-22T03:36:15.366810+0000","last_peered":"2024-09-22T03:36:15.366810+0000","last_clean":"2024-09-22T03:36:15.366810+0000","last_became_active":"2024-09-22T03:36:13.426104+0000","last_became_peered":"2024-09-22T03:36:13.426104+0000","last_unstale":"2024-09-22T03:36:15.366810+0000","last_undegraded":"2024-09-22T03:36:15.366810+0000","last_fullsized":"2024-09-22T03:36:15.366810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364775+0000","last_change":"2024-09-22T03:36:15.367608+0000","last_active":"2024-09-22T03:36:15.364775+0000","last_peered":"2024-09-22T03:36:15.364775+0000","last_clean":"2024-09-22T03:36:15.364775+0000","last_became_active":"2024-09-22T03:36:13.419211+0000","last_became_peered":"2024-09-22T03:36:13.419211+0000","last_unstale":"2024-09-22T03:36:15.364775+0000","last_undegraded":"2024-09-22T03:36:15.364775+0000","last_fullsized":"2024-09-22T03:36:15.364775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364703+0000","last_change":"2024-09-22T03:36:15.367640+0000","last_active":"2024-09-22T03:36:15.364703+0000","last_peered":"2024-09-22T03:36:15.364703+0000","last_clean":"2024-09-22T03:36:15.364703+0000","last_became_active":"2024-09-22T03:36:13.412782+0000","last_became_peered":"2024-09-22T03:36:13.412782+0000","last_unstale":"2024-09-22T03:36:15.364703+0000","last_undegraded":"2024-09-22T03:36:15.364703+0000","last_fullsized":"2024-09-22T03:36:15.364703+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366734+0000","last_change":"2024-09-22T03:36:15.369125+0000","last_active":"2024-09-22T03:36:15.366734+0000","last_peered":"2024-09-22T03:36:15.366734+0000","last_clean":"2024-09-22T03:36:15.366734+0000","last_became_active":"2024-09-22T03:36:13.428256+0000","last_became_peered":"2024-09-22T03:36:13.428256+0000","last_unstale":"2024-09-22T03:36:15.366734+0000","last_undegraded":"2024-09-22T03:36:15.366734+0000","last_fullsized":"2024-09-22T03:36:15.366734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.383445+0000","last_change":"2024-09-22T03:36:15.383445+0000","last_active":"2024-09-22T03:36:15.383445+0000","last_peered":"2024-09-22T03:36:15.383445+0000","last_clean":"2024-09-22T03:36:15.383445+0000","last_became_active":"2024-09-22T03:36:13.415632+0000","last_became_peered":"2024-09-22T03:36:13.415632+0000","last_unstale":"2024-09-22T03:36:15.383445+0000","last_undegraded":"2024-09-22T03:36:15.383445+0000","last_fullsized":"2024-09-22T03:36:15.383445+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.373934+0000","last_change":"2024-09-22T03:36:15.375087+0000","last_active":"2024-09-22T03:36:15.373934+0000","last_peered":"2024-09-22T03:36:15.373934+0000","last_clean":"2024-09-22T03:36:15.373934+0000","last_became_active":"2024-09-22T03:36:13.387907+0000","last_became_peered":"2024-09-22T03:36:13.387907+0000","last_unstale":"2024-09-22T03:36:15.373934+0000","last_undegraded":"2024-09-22T03:36:15.373934+0000","last_fullsized":"2024-09-22T03:36:15.373934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.371951+0000","last_change":"2024-09-22T03:36:15.373663+0000","last_active":"2024-09-22T03:36:15.371951+0000","last_peered":"2024-09-22T03:36:15.371951+0000","last_clean":"2024-09-22T03:36:15.371951+0000","last_became_active":"2024-09-22T03:36:13.383785+0000","last_became_peered":"2024-09-22T03:36:13.383785+0000","last_unstale":"2024-09-22T03:36:15.371951+0000","last_undegraded":"2024-09-22T03:36:15.371951+0000","last_fullsized":"2024-09-22T03:36:15.371951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366079+0000","last_change":"2024-09-22T03:36:15.368053+0000","last_active":"2024-09-22T03:36:15.366079+0000","last_peered":"2024-09-22T03:36:15.366079+0000","last_clean":"2024-09-22T03:36:15.366079+0000","last_became_active":"2024-09-22T03:36:13.394346+0000","last_became_peered":"2024-09-22T03:36:13.394346+0000","last_unstale":"2024-09-22T03:36:15.366079+0000","last_undegraded":"2024-09-22T03:36:15.366079+0000","last_fullsized":"2024-09-22T03:36:15.366079+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364384+0000","last_change":"2024-09-22T03:36:11.376929+0000","last_active":"2024-09-22T03:36:15.364384+0000","last_peered":"2024-09-22T03:36:15.364384+0000","last_clean":"2024-09-22T03:36:15.364384+0000","last_became_active":"2024-09-22T03:36:11.371229+0000","last_became_peered":"2024-09-22T03:36:11.371229+0000","last_unstale":"2024-09-22T03:36:15.364384+0000","last_undegraded":"2024-09-22T03:36:15.364384+0000","last_fullsized":"2024-09-22T03:36:15.364384+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:10.327972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:10.327972+0000","last_clean_scrub_stamp":"2024-09-22T03:36:10.327972+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":464,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297264,"statfs":{"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1624,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296104,"statfs":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1368,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296360,"statfs":{"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:36:17.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-22T03:36:17.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:17.252+0000 7fdb1804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:17.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:17.257+0000 7fdb1804f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:17.430 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:17.431 INFO:teuthology.orchestra.run.smithi117.stderr:dumped all 2024-09-22T03:36:17.442 INFO:teuthology.orchestra.run.smithi117.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-22T03:36:16.460300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":156893184,"kb_used":3456,"kb_used_data":3456,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":156889728,"statfs":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366810+0000","last_change":"2024-09-22T03:36:15.369055+0000","last_active":"2024-09-22T03:36:15.366810+0000","last_peered":"2024-09-22T03:36:15.366810+0000","last_clean":"2024-09-22T03:36:15.366810+0000","last_became_active":"2024-09-22T03:36:13.426104+0000","last_became_peered":"2024-09-22T03:36:13.426104+0000","last_unstale":"2024-09-22T03:36:15.366810+0000","last_undegraded":"2024-09-22T03:36:15.366810+0000","last_fullsized":"2024-09-22T03:36:15.366810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364775+0000","last_change":"2024-09-22T03:36:15.367608+0000","last_active":"2024-09-22T03:36:15.364775+0000","last_peered":"2024-09-22T03:36:15.364775+0000","last_clean":"2024-09-22T03:36:15.364775+0000","last_became_active":"2024-09-22T03:36:13.419211+0000","last_became_peered":"2024-09-22T03:36:13.419211+0000","last_unstale":"2024-09-22T03:36:15.364775+0000","last_undegraded":"2024-09-22T03:36:15.364775+0000","last_fullsized":"2024-09-22T03:36:15.364775+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364703+0000","last_change":"2024-09-22T03:36:15.367640+0000","last_active":"2024-09-22T03:36:15.364703+0000","last_peered":"2024-09-22T03:36:15.364703+0000","last_clean":"2024-09-22T03:36:15.364703+0000","last_became_active":"2024-09-22T03:36:13.412782+0000","last_became_peered":"2024-09-22T03:36:13.412782+0000","last_unstale":"2024-09-22T03:36:15.364703+0000","last_undegraded":"2024-09-22T03:36:15.364703+0000","last_fullsized":"2024-09-22T03:36:15.364703+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366734+0000","last_change":"2024-09-22T03:36:15.369125+0000","last_active":"2024-09-22T03:36:15.366734+0000","last_peered":"2024-09-22T03:36:15.366734+0000","last_clean":"2024-09-22T03:36:15.366734+0000","last_became_active":"2024-09-22T03:36:13.428256+0000","last_became_peered":"2024-09-22T03:36:13.428256+0000","last_unstale":"2024-09-22T03:36:15.366734+0000","last_undegraded":"2024-09-22T03:36:15.366734+0000","last_fullsized":"2024-09-22T03:36:15.366734+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.383445+0000","last_change":"2024-09-22T03:36:15.383445+0000","last_active":"2024-09-22T03:36:15.383445+0000","last_peered":"2024-09-22T03:36:15.383445+0000","last_clean":"2024-09-22T03:36:15.383445+0000","last_became_active":"2024-09-22T03:36:13.415632+0000","last_became_peered":"2024-09-22T03:36:13.415632+0000","last_unstale":"2024-09-22T03:36:15.383445+0000","last_undegraded":"2024-09-22T03:36:15.383445+0000","last_fullsized":"2024-09-22T03:36:15.383445+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.373934+0000","last_change":"2024-09-22T03:36:15.375087+0000","last_active":"2024-09-22T03:36:15.373934+0000","last_peered":"2024-09-22T03:36:15.373934+0000","last_clean":"2024-09-22T03:36:15.373934+0000","last_became_active":"2024-09-22T03:36:13.387907+0000","last_became_peered":"2024-09-22T03:36:13.387907+0000","last_unstale":"2024-09-22T03:36:15.373934+0000","last_undegraded":"2024-09-22T03:36:15.373934+0000","last_fullsized":"2024-09-22T03:36:15.373934+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.371951+0000","last_change":"2024-09-22T03:36:15.373663+0000","last_active":"2024-09-22T03:36:15.371951+0000","last_peered":"2024-09-22T03:36:15.371951+0000","last_clean":"2024-09-22T03:36:15.371951+0000","last_became_active":"2024-09-22T03:36:13.383785+0000","last_became_peered":"2024-09-22T03:36:13.383785+0000","last_unstale":"2024-09-22T03:36:15.371951+0000","last_undegraded":"2024-09-22T03:36:15.371951+0000","last_fullsized":"2024-09-22T03:36:15.371951+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.366079+0000","last_change":"2024-09-22T03:36:15.368053+0000","last_active":"2024-09-22T03:36:15.366079+0000","last_peered":"2024-09-22T03:36:15.366079+0000","last_clean":"2024-09-22T03:36:15.366079+0000","last_became_active":"2024-09-22T03:36:13.394346+0000","last_became_peered":"2024-09-22T03:36:13.394346+0000","last_unstale":"2024-09-22T03:36:15.366079+0000","last_undegraded":"2024-09-22T03:36:15.366079+0000","last_fullsized":"2024-09-22T03:36:15.366079+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:12.335746+0000","last_clean_scrub_stamp":"2024-09-22T03:36:12.335746+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-22T03:36:15.364384+0000","last_change":"2024-09-22T03:36:11.376929+0000","last_active":"2024-09-22T03:36:15.364384+0000","last_peered":"2024-09-22T03:36:15.364384+0000","last_clean":"2024-09-22T03:36:15.364384+0000","last_became_active":"2024-09-22T03:36:11.371229+0000","last_became_peered":"2024-09-22T03:36:11.371229+0000","last_unstale":"2024-09-22T03:36:15.364384+0000","last_undegraded":"2024-09-22T03:36:15.364384+0000","last_fullsized":"2024-09-22T03:36:15.364384+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-22T03:36:10.327972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-22T03:36:10.327972+0000","last_clean_scrub_stamp":"2024-09-22T03:36:10.327972+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":160658620416,"available":160655081472,"internally_reserved":0,"allocated":3538944,"data_stored":3538944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":1}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":464,"kb_used_data":464,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52297264,"statfs":{"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1624,"kb_used_data":1624,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296104,"statfs":{"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705672,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52297728,"kb_used":1368,"kb_used_data":1368,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":52296360,"statfs":{"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":53552873472,"available":53551472640,"internally_reserved":0,"allocated":1400832,"data_stored":1400832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":53552873472,"available":53551210496,"internally_reserved":0,"allocated":1662976,"data_stored":1662976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":53552873472,"available":53552398336,"internally_reserved":0,"allocated":475136,"data_stored":475136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-22T03:36:17.443 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-22T03:36:17.443 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-22T03:36:17.443 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-22T03:36:17.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-22T03:36:17.547 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:17.543+0000 7fbb2ab57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:17.548 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:36:17.547+0000 7fbb2ab57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:36:17.756 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:17.757 INFO:teuthology.orchestra.run.smithi117.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-22T03:36:17.769 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-22T03:36:17.769 INFO:teuthology.run_tasks:Running task cbt... 2024-09-22T03:36:17.782 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi117.front.sepia.ceph.com', 'osds': ['ubuntu@smithi117.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi117.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi117.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi117.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-09-22T03:36:17.782 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-09-22T03:36:17.802 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:36:17.803 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:36:17.853 DEBUG:teuthology.orchestra.run.smithi117:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-09-22T03:36:17.908 INFO:teuthology.orchestra.run.smithi117.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-09-22T03:36:18.213 INFO:tasks.cbt:Installing collectl 2024-09-22T03:36:18.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-09-22T03:36:18.319 INFO:teuthology.orchestra.run.smithi117.stderr:--2024-09-22 03:36:18-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-09-22T03:36:18.468 INFO:teuthology.orchestra.run.smithi117.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2024-09-22T03:36:18.477 INFO:teuthology.orchestra.run.smithi117.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-09-22T03:36:18.701 INFO:teuthology.orchestra.run.smithi117.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:36:18.701 INFO:teuthology.orchestra.run.smithi117.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75CynsFC_y-VAElr5k3k6uSFhpX7PRMM3n9yqcG65p1HfvMBL71r3ZPIX1RgCs3xQB0Sd91bngboaeJqKDrO9smWLQ%3D%3D&use_mirror=phoenixnap&r= [following] 2024-09-22T03:36:18.701 INFO:teuthology.orchestra.run.smithi117.stderr:--2024-09-22 03:36:18-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABm75CynsFC_y-VAElr5k3k6uSFhpX7PRMM3n9yqcG65p1HfvMBL71r3ZPIX1RgCs3xQB0Sd91bngboaeJqKDrO9smWLQ%3D%3D&use_mirror=phoenixnap&r= 2024-09-22T03:36:18.701 INFO:teuthology.orchestra.run.smithi117.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-09-22T03:36:18.774 INFO:teuthology.orchestra.run.smithi117.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-09-22T03:36:19.065 INFO:teuthology.orchestra.run.smithi117.stderr:HTTP request sent, awaiting response... 302 Found 2024-09-22T03:36:19.065 INFO:teuthology.orchestra.run.smithi117.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 [following] 2024-09-22T03:36:19.065 INFO:teuthology.orchestra.run.smithi117.stderr:--2024-09-22 03:36:19-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?viasf=1 2024-09-22T03:36:19.065 INFO:teuthology.orchestra.run.smithi117.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2024-09-22T03:36:19.127 INFO:teuthology.orchestra.run.smithi117.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2024-09-22T03:36:19.418 INFO:teuthology.orchestra.run.smithi117.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-22T03:36:19.418 INFO:teuthology.orchestra.run.smithi117.stderr:Length: 620693 (606K) [application/x-gzip] 2024-09-22T03:36:19.418 INFO:teuthology.orchestra.run.smithi117.stderr:Saving to: ‘collectl.tar.gz’ 2024-09-22T03:36:19.418 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:36:19.507 INFO:teuthology.orchestra.run.smithi117.stderr: 0K .......... .......... .......... .......... .......... 8% 557K 1s 2024-09-22T03:36:19.569 INFO:teuthology.orchestra.run.smithi117.stderr: 50K .......... .......... .......... .......... .......... 16% 814K 1s 2024-09-22T03:36:19.569 INFO:teuthology.orchestra.run.smithi117.stderr: 100K .......... .......... .......... .......... .......... 24% 125M 0s 2024-09-22T03:36:19.602 INFO:teuthology.orchestra.run.smithi117.stderr: 150K .......... .......... .......... .......... .......... 32% 1.49M 0s 2024-09-22T03:36:19.632 INFO:teuthology.orchestra.run.smithi117.stderr: 200K .......... .......... .......... .......... .......... 41% 1.72M 0s 2024-09-22T03:36:19.632 INFO:teuthology.orchestra.run.smithi117.stderr: 250K .......... .......... .......... .......... .......... 49% 182M 0s 2024-09-22T03:36:19.632 INFO:teuthology.orchestra.run.smithi117.stderr: 300K .......... .......... .......... .......... .......... 57% 303M 0s 2024-09-22T03:36:19.663 INFO:teuthology.orchestra.run.smithi117.stderr: 350K .......... .......... .......... .......... .......... 65% 1.51M 0s 2024-09-22T03:36:19.664 INFO:teuthology.orchestra.run.smithi117.stderr: 400K .......... .......... .......... .......... .......... 74% 233M 0s 2024-09-22T03:36:19.693 INFO:teuthology.orchestra.run.smithi117.stderr: 450K .......... .......... .......... .......... .......... 82% 1.72M 0s 2024-09-22T03:36:19.693 INFO:teuthology.orchestra.run.smithi117.stderr: 500K .......... .......... .......... .......... .......... 90% 296M 0s 2024-09-22T03:36:19.693 INFO:teuthology.orchestra.run.smithi117.stderr: 550K .......... .......... .......... .......... .......... 98% 332M 0s 2024-09-22T03:36:19.693 INFO:teuthology.orchestra.run.smithi117.stderr: 600K ...... 100% 11.4T=0.3s 2024-09-22T03:36:19.693 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:36:19.694 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22 03:36:19 (2.16 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-09-22T03:36:19.694 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:36:19.727 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/ 2024-09-22T03:36:19.727 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/collectl.conf 2024-09-22T03:36:19.727 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/ 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/SlowProc.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/WhySummary.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Socket.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/colmux.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Tutorial.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Gexpr.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Import.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/style.css 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/FileNaming.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/TheMath.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Environmental.html 2024-09-22T03:36:19.728 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Process.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/HiResTime.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/slub.jpg 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Lustre.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Documentation.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Inodes.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Graphite.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/InputFiles.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Export.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Interrupts.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Startup.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Performance.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Messages.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Logging.html 2024-09-22T03:36:19.729 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Data-detail.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/CPUs.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Matrix.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Data-brief.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Misc.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Examples.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/OpenStack.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Data.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Exceptions.html 2024-09-22T03:36:19.730 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Playback.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Hello.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Sockets.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Network.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Infiniband.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Memory.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/Disks.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/service/ 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/service/collectl.service 2024-09-22T03:36:19.731 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/initd/ 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/initd/collectl-suse 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/initd/collectl-debian 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/initd/collectl-generic 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/initd/collectl 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/misc.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/gexpr.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/statsd.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/envrules.std 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/hello.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/vmsum.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/proctree.ph 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/README-WINDOWS 2024-09-22T03:36:19.732 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/formatit.ph 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/colmux 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/INSTALL 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/graphite.ph 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/RELEASE-collectl 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/vmstat.ph 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/README 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/man1/ 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/man1/collectl.1 2024-09-22T03:36:19.734 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/man1/colmux.1 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/lexpr.ph 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/client.pl 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/collectl 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/COPYING 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/vnet.ph 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/GPL 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/UNINSTALL 2024-09-22T03:36:19.735 INFO:teuthology.orchestra.run.smithi117.stdout:collectl-4.3.1/ARTISTIC 2024-09-22T03:36:19.789 INFO:teuthology.orchestra.run.smithi117.stderr:./INSTALL: line 90: chkconfig: command not found 2024-09-22T03:36:19.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-09-22T03:36:20.380 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:03:20 ago on Sun 22 Sep 2024 03:33:00 AM UTC. 2024-09-22T03:36:20.475 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-09-22T03:36:20.477 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-09-22T03:36:20.525 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout: librbd-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 23 k 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout: perf x86_64 5.14.0-508.el9 appstream 2.2 M 2024-09-22T03:36:20.526 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout: libradospp-devel x86_64 2:19.1.1-373.g4e45e6b4.el9 ceph 30 k 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout:Install 5 Packages 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.4 M 2024-09-22T03:36:20.527 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 11 M 2024-09-22T03:36:20.528 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-09-22T03:36:20.890 INFO:teuthology.orchestra.run.smithi117.stdout:(1/5): libradospp-devel-19.1.1-373.g4e45e6b4.el 180 kB/s | 30 kB 00:00 2024-09-22T03:36:20.915 INFO:teuthology.orchestra.run.smithi117.stdout:(2/5): librbd-devel-19.1.1-373.g4e45e6b4.el9.x8 120 kB/s | 23 kB 00:00 2024-09-22T03:36:20.974 INFO:teuthology.orchestra.run.smithi117.stdout:(3/5): perf-5.14.0-508.el9.x86_64.rpm 8.8 MB/s | 2.2 MB 00:00 2024-09-22T03:36:21.024 INFO:teuthology.orchestra.run.smithi117.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 121 kB/s | 13 kB 00:00 2024-09-22T03:36:21.049 INFO:teuthology.orchestra.run.smithi117.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 747 kB/s | 118 kB 00:00 2024-09-22T03:36:21.052 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-09-22T03:36:21.052 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.6 MB/s | 2.4 MB 00:00 2024-09-22T03:36:21.081 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-09-22T03:36:21.102 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-09-22T03:36:21.103 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-09-22T03:36:21.227 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-09-22T03:36:21.228 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-09-22T03:36:21.580 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-09-22T03:36:21.651 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-09-22T03:36:21.691 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-09-22T03:36:21.730 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradospp-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 3/5 2024-09-22T03:36:22.060 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librbd-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 4/5 2024-09-22T03:36:22.143 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:36:22.946 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: perf-5.14.0-508.el9.x86_64 5/5 2024-09-22T03:36:22.946 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradospp-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 1/5 2024-09-22T03:36:22.946 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2/5 2024-09-22T03:36:22.946 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : perf-5.14.0-508.el9.x86_64 3/5 2024-09-22T03:36:22.946 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: libradospp-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: librbd-devel-2:19.1.1-373.g4e45e6b4.el9.x86_64 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: pdsh-2.34-7.el9.x86_64 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: perf-5.14.0-508.el9.x86_64 2024-09-22T03:36:23.272 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-09-22T03:36:23.273 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-09-22T03:36:23.642 INFO:tasks.cbt:benchmark: radosbench 2024-09-22T03:36:23.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-09-22T03:36:23.667 DEBUG:teuthology.orchestra.run.smithi117:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-09-22T03:36:24.016 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - Settings.cluster: 2024-09-22T03:36:24.016 INFO:teuthology.orchestra.run.smithi117.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'clients': ['ubuntu@smithi117.front.sepia.ceph.com'], 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'head': 'ubuntu@smithi117.front.sepia.ceph.com', 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'is_teuthology': True, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'iterations': 1, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'mons': ['ubuntu@smithi117.front.sepia.ceph.com'], 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'osds': ['ubuntu@smithi117.front.sepia.ceph.com'], 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'osds_per_node': 3, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'pid_dir': '/var/run/ceph', 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'pgp_size': 256, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'replication': 'replicated'}}, 2024-09-22T03:36:24.017 INFO:teuthology.orchestra.run.smithi117.stderr: 'rebuild_every_test': False, 2024-09-22T03:36:24.018 INFO:teuthology.orchestra.run.smithi117.stderr: 'rgws': {'ubuntu@smithi117.front.sepia.ceph.com': None}, 2024-09-22T03:36:24.018 INFO:teuthology.orchestra.run.smithi117.stderr: 'tmp_dir': '/tmp/cbt', 2024-09-22T03:36:24.018 INFO:teuthology.orchestra.run.smithi117.stderr: 'use_existing': True, 2024-09-22T03:36:24.018 INFO:teuthology.orchestra.run.smithi117.stderr: 'user': 'ubuntu'} 2024-09-22T03:36:24.018 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T03:36:24.035 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-09-22T03:36:24.037 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2024-09-22T03:36:24.040 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:36:24.041 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2024-09-22T03:36:24.044 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:36:24.044 INFO:teuthology.orchestra.run.smithi117.stderr:03:36:24 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:37:24.106 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:37:24.175 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:37:24.227 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-09-22T03:37:24.228 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-09-22T03:37:24.230 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2024-09-22T03:37:24.244 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi117.front.sepia.ceph.com ; done 2024-09-22T03:37:24.248 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-09-22T03:37:24.277 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - WARNING - cbt - sudo killall -9 rados 2024-09-22T03:37:24.277 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - WARNING - cbt - error 1 seen, continuing anyway... 2024-09-22T03:37:24.277 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - INFO - cbt - Results dir: /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - INFO - cbt - Running benchmark /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:{'concurrent_ops': 4, 'concurrent_procs': 2, 'pool_profile': 'replicated', 'time': 60, 'write_only': True, 'op_size': 4096, 'pool_monitoring_list': 'collectl', 'run_monitoring_list': 'collectl', 'benchmark': 'radosbench', 'iteration': 0} == iteration 0 == 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T03:37:24.296 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-09-22T03:37:24.309 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2024-09-22T03:37:24.312 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:37:24.314 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2024-09-22T03:37:24.317 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:37:24.317 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:37:24.317 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-09-22T03:37:24.609 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:24 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-09-22T03:37:25.528 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-09-22T03:37:26.538 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:26 - INFO - cbt - Checking Health after pool creation. 2024-09-22T03:37:31.544 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:37:31.855 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:31 - INFO - cbt - Final Pool Health Check. 2024-09-22T03:37:36.860 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:36 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-09-22T03:37:37.179 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:37:37.217 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:37:37.265 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-09-22T03:37:37.266 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-09-22T03:37:37.268 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-09-22T03:37:37.504 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-09-22T03:37:38.082 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-09-22T03:37:38.446 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2024-09-22T03:37:38.492 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2024-09-22T03:37:39.369 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2024-09-22T03:37:39.372 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-09-22T03:37:39.374 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2024-09-22T03:37:39.377 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - PerfMonitoring: in local_node 2024-09-22T03:37:39.378 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-09-22T03:37:39.378 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - INFO - cbt - Running radosbench write test. 2024-09-22T03:37:39.378 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2024-09-22T03:37:39.379 INFO:teuthology.orchestra.run.smithi117.stderr:03:37:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2024-09-22T03:38:40.475 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-09-22T03:38:40.559 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-09-22T03:38:40.605 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2024-09-22T03:38:40.606 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2024-09-22T03:38:40.608 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2024-09-22T03:38:40.958 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2024-09-22T03:38:40.971 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi117.front.sepia.ceph.com ; done 2024-09-22T03:38:41.017 INFO:teuthology.orchestra.run.smithi117.stderr:03:38:41 - INFO - cbt - Convert results to json format. 2024-09-22T03:38:41.033 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-09-22T03:38:41.042 DEBUG:teuthology.orchestra.run.smithi117:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-09-22T03:38:41.244 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-09-22T03:38:41.244 INFO:tasks.cbt_performance:reading cbt preferences 2024-09-22T03:38:41.244 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-09-22T03:38:41.334 INFO:teuthology.orchestra.run.smithi117.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi117.front.sepia.ceph.com 2024-09-22T03:38:41.334 INFO:teuthology.orchestra.run.smithi117.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi117.front.sepia.ceph.com 2024-09-22T03:38:41.334 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:38:41.335 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi117.front.sepia.ceph.com of=/dev/stdout 2024-09-22T03:38:41.355 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T03:38:41.356 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-09-22T03:38:41.411 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T03:38:41.411 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T03:38:41.411 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-09-22T03:38:41.411 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi117.front.sepia.ceph.com of=/dev/stdout 2024-09-22T03:38:41.466 INFO:tasks.cbt_performance:reading total cpu cycles 2024-09-22T03:38:41.466 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2024-09-22T03:38:41.521 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-09-22T03:38:41.521 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-09-22T03:38:41.530 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T03:38:41.539 INFO:tasks.cbt_performance:Data inserted successfully. 2024-09-22T03:38:41.539 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-22T03:38:41.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-22T03:38:41.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:38:41.657+0000 7f06cfb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:38:41.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-09-22T03:38:41.665+0000 7f06cfb57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-09-22T03:38:41.855 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-22T03:38:41.855 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-22T03:38:41.855 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-22T03:38:41.855 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:38:44.385 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-09-22T03:38:47.957 INFO:tasks.ceph.osd.0:Stopped 2024-09-22T03:38:47.957 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-22T03:38:47.957 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:38:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-09-22T03:38:49.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-09-22T03:38:54.058 INFO:tasks.ceph.osd.1:Stopped 2024-09-22T03:38:54.059 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-22T03:38:54.059 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:38:54.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-22T03:38:54.890 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-09-22T03:39:00.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2024-09-22T03:39:00.162 INFO:tasks.ceph.osd.2:Stopped 2024-09-22T03:39:00.162 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-22T03:39:00.163 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-22T03:39:00.163 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:39:00.200 INFO:tasks.ceph.mgr.x:Stopped 2024-09-22T03:39:00.200 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-22T03:39:00.200 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-22T03:39:00.200 INFO:teuthology.orchestra.run:waiting for 300 2024-09-22T03:39:00.254 INFO:tasks.ceph.mon.a:Stopped 2024-09-22T03:39:00.254 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-22T03:39:00.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-22T03:39:00.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:39:00.340 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-22T03:39:00.583 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:39:00.583 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-22T03:39:00.644 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:39:00.644 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-22T03:39:00.726 INFO:tasks.ceph:Archiving mon data... 2024-09-22T03:39:00.727 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785/data/mon.a.tgz 2024-09-22T03:39:00.727 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-09-22T03:39:00.744 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.HmAy7vUerR 2024-09-22T03:39:00.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.HmAy7vUerR 2024-09-22T03:39:00.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.HmAy7vUerR 2024-09-22T03:39:00.991 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.HmAy7vUerR is 181KB 2024-09-22T03:39:01.043 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.HmAy7vUerR 2024-09-22T03:39:01.057 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-22T03:39:01.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-22T03:39:01.192 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-22T03:39:01.192 INFO:tasks.ceph:Archiving crash dumps... 2024-09-22T03:39:01.194 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785/remote/smithi117/crash 2024-09-22T03:39:01.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-22T03:39:01.256 INFO:tasks.ceph:Compressing logs... 2024-09-22T03:39:01.256 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-22T03:39:01.330 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-22T03:39:01.330 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29051.log 2024-09-22T03:39:01.330 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28812.log 2024-09-22T03:39:01.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.29051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30889.log 2024-09-22T03:39:01.331 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.28812.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.30889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28574.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.30889.log.gz 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30740.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.28574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29434.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29642.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.30740.log: /var/log/ceph/ceph-client.admin.29434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29641.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29728.log 2024-09-22T03:39:01.332 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29515.log 2024-09-22T03:39:01.333 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30888.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29602.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29255.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31127.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29021.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28662.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29812.log 2024-09-22T03:39:01.334 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29514.log 2024-09-22T03:39:01.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29770.log 2024-09-22T03:39:01.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30675.log 2024-09-22T03:39:01.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29199.log 2024-09-22T03:39:01.335 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29392.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30801.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29350.log 2024-09-22T03:39:01.336 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28856.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27620.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29516.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28772.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29299.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30777.log 2024-09-22T03:39:01.338 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30824.log 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.29641.log: /var/log/ceph/ceph-client.admin.29515.log: /var/log/ceph/ceph-client.admin.30888.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28574.log.gz 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.29642.log: /var/log/ceph/ceph-client.admin.30777.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.30824.log: /var/log/ceph/ceph-client.admin.31127.log: /var/log/ceph/ceph-client.admin.29299.log: /var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.log: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz/var/log/ceph/ceph-mgr.x.log: 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30740.log.gz 49.6% 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31127.log.gz 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.29728.log: /var/log/ceph/ceph-client.admin.29021.log: /var/log/ceph/ceph-client.admin.29602.log: /var/log/ceph/ceph-client.admin.29255.log: /var/log/ceph/ceph-client.admin.29812.log: /var/log/ceph/ceph-client.admin.28856.log: /var/log/ceph/ceph.tmp-client.admin.27620.log: /var/log/ceph/ceph-client.admin.29770.log: /var/log/ceph/ceph-client.admin.28662.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.30888.log.gz 2024-09-22T03:39:01.352 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.29199.log: 84.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-22T03:39:01.354 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.28772.log: /var/log/ceph/ceph-client.admin.29392.log: /var/log/ceph/ceph-client.admin.30801.log: /var/log/ceph/ceph-client.admin.29350.log: /var/log/ceph/ceph-client.admin.29514.log: /var/log/ceph/ceph-client.admin.30675.log: /var/log/ceph/ceph-client.admin.29516.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30777.log.gz 2024-09-22T03:39:01.355 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28662.log.gz 2024-09-22T03:39:01.355 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-22T03:39:01.363 INFO:teuthology.orchestra.run.smithi117.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-22T03:39:01.367 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29199.log.gz 2024-09-22T03:39:01.367 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29255.log.gz 2024-09-22T03:39:01.367 INFO:teuthology.orchestra.run.smithi117.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.29299.log.gz 2024-09-22T03:39:01.367 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29350.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29392.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29434.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29515.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29514.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29516.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29602.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29641.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29642.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29728.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29770.log.gz 2024-09-22T03:39:01.368 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29812.log.gz 2024-09-22T03:39:01.370 INFO:teuthology.orchestra.run.smithi117.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30675.log.gz 2024-09-22T03:39:01.371 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-22T03:39:01.380 INFO:teuthology.orchestra.run.smithi117.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27620.log.gz 2024-09-22T03:39:01.383 INFO:teuthology.orchestra.run.smithi117.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28812.log.gz 2024-09-22T03:39:01.383 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28856.log.gz 2024-09-22T03:39:01.387 INFO:teuthology.orchestra.run.smithi117.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-22T03:39:01.389 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28772.log.gz 2024-09-22T03:39:01.391 INFO:teuthology.orchestra.run.smithi117.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29021.log.gz 2024-09-22T03:39:01.392 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29051.log.gz 2024-09-22T03:39:01.400 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30801.log.gz 2024-09-22T03:39:01.400 INFO:teuthology.orchestra.run.smithi117.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30824.log.gz 2024-09-22T03:39:01.418 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-22T03:39:01.420 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-09-22T03:39:01.420 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m0.110s 2024-09-22T03:39:01.420 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m0.041s 2024-09-22T03:39:01.420 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.030s 2024-09-22T03:39:01.421 INFO:tasks.ceph:Archiving logs... 2024-09-22T03:39:01.421 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785/remote/smithi117/log 2024-09-22T03:39:01.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-22T03:39:01.541 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-22T03:39:01.551 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-22T03:39:01.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-22T03:39:01.612 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-22T03:39:01.620 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-22T03:39:01.620 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-22T03:39:01.648 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 -178us[ -174us] +/- 56ms 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 49 +657us[ +661us] +/- 80ms 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -299us[ -295us] +/- 65ms 2024-09-22T03:39:01.716 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-22T03:39:01.731 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-22T03:39:01.740 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-22T03:39:01.741 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-22T03:39:01.749 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-22T03:39:01.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-22T03:39:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:39:01.911 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-22T03:39:01.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-22T03:39:01.929 INFO:teuthology.task.internal:Duration was 603.962329 seconds 2024-09-22T03:39:01.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-22T03:39:01.938 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-22T03:39:01.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-22T03:39:02.078 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-22T03:39:02.222 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-22T03:39:02.222 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-09-22T03:39:02.222 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-22T03:39:02.242 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-22T03:39:02.242 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-22T03:39:02.307 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-22T03:39:02.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-22T03:39:02.457 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-22T03:39:02.467 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-22T03:39:02.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-22T03:39:02.503 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-22T03:39:02.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:39:02.587 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-09-22T03:39:02.609 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-22T03:39:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-22T03:39:02.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-22T03:39:02.633 INFO:teuthology.task.internal:Transferring archived files... 2024-09-22T03:39:02.633 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:56:02-crimson-rados-squid-distro-crimson-smithi/7914785/remote/smithi117 2024-09-22T03:39:02.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-22T03:39:02.739 INFO:teuthology.task.internal:Removing archive directory... 2024-09-22T03:39:02.739 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-22T03:39:02.772 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-22T03:39:02.791 INFO:teuthology.task.internal:Not uploading archives. 2024-09-22T03:39:02.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-22T03:39:02.803 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-22T03:39:02.803 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-22T03:39:02.827 INFO:teuthology.orchestra.run.smithi117.stdout: 265885 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 22 03:39 /home/ubuntu/cephtest 2024-09-22T03:39:02.828 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-22T03:39:02.838 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-22T03:39:02.858 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=squid&var-job_id_selected=7914785 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore settings/optimized workloads/sample_radosbench} duration: 603.9623286724091 flavor: crimson owner: scheduled_teuthology@teuthology success: true 2024-09-22T03:39:02.858 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-22T03:39:02.939 INFO:teuthology.run:pass