2023-12-19T10:31:51.568 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-19T10:31:51.568 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733 --owner scheduled_matan@teuthology --name matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} -- /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733/orig.config.yaml 2023-12-19T10:31:51.611 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T10:31:51.700 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733 branch: wip-matanb-crimson-testing-misc description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} email: null first_in_suite: false job_id: '7496733' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-misc ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d workunit: branch: wip-matanb-crimson-testing-misc env: CRIMSON_COMPAT: '1' sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6360 sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-testing-misc suite_path: /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa suite_relpath: qa suite_repo: https://github.com/Matan-B/ceph.git suite_sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d targets: smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP65EXxXwkBSVWjXgoG0j74AGak5Q+rkRAN+nMX7H7VzMMSc+GAKyqBuvv3Qq1V0m6FjaKh8VWFprIHSO0lHglw= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-19_10:13:57 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 2023-12-19T10:31:51.700 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa; will attempt to use it 2023-12-19T10:31:51.701 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa/tasks 2023-12-19T10:31:51.701 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-19T10:31:51.702 INFO:teuthology.task.internal:Checking packages... 2023-12-19T10:31:51.727 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '729869e4a72e72e756b33d15b6f2fbf113fa1a2d' 2023-12-19T10:31:51.727 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-19T10:31:51.727 INFO:teuthology.packaging:ref: None 2023-12-19T10:31:51.727 INFO:teuthology.packaging:tag: None 2023-12-19T10:31:51.727 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-misc 2023-12-19T10:31:51.727 INFO:teuthology.packaging:sha1: 729869e4a72e72e756b33d15b6f2fbf113fa1a2d 2023-12-19T10:31:51.727 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-misc 2023-12-19T10:31:51.866 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-231.g729869e4 2023-12-19T10:31:51.867 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-19T10:31:51.874 INFO:teuthology.task.internal:no buildpackages task found 2023-12-19T10:31:51.874 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-19T10:31:51.886 INFO:teuthology.task.internal:Saving configuration 2023-12-19T10:31:51.904 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-19T10:31:51.911 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-19T10:31:51.936 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-19 10:27:22.625693', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP65EXxXwkBSVWjXgoG0j74AGak5Q+rkRAN+nMX7H7VzMMSc+GAKyqBuvv3Qq1V0m6FjaKh8VWFprIHSO0lHglw='} 2023-12-19T10:31:51.936 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-19T10:31:51.942 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-12-19T10:31:51.942 INFO:teuthology.run_tasks:Running task console_log... 2023-12-19T10:31:51.984 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5afc02de50>, signals=[15]) 2023-12-19T10:31:51.985 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-19T10:31:51.991 INFO:teuthology.task.internal:Opening connections... 2023-12-19T10:31:51.991 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:31:51.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T10:31:52.137 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-19T10:31:52.146 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2023-12-19T10:31:52.162 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-12-19T10:31:52.163 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2023-12-19T10:31:52.220 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-19T10:31:52.221 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-19T10:31:52.222 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-12-19T10:31:52.246 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-19T10:31:52.255 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-19T10:31:52.262 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-19T10:31:52.263 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2023-12-19T10:31:52.279 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-19T10:31:52.286 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-19T10:31:52.286 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2023-12-19T10:31:52.345 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-19T10:31:52.378 INFO:teuthology.run_tasks:Running task kernel... 2023-12-19T10:31:52.392 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-19T10:31:52.392 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-19T10:31:52.392 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-19T10:31:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T10:31:52.409 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-19T10:31:52.465 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2023-12-19T10:31:52.466 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2023-12-19T10:31:52.466 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2023-12-19T10:31:54.479 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-19T10:31:55.837 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-12-19T10:31:57.903 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 7.9 MB 00:01 2023-12-19T10:32:03.103 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 18 MB 00:04 2023-12-19T10:32:08.020 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 5.8 MB 00:01 2023-12-19T10:32:09.471 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2023-12-19T10:32:10.806 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-12-19T10:32:15.664 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2023-12-19T10:32:17.870 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-19T10:32:17.871 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-19T10:32:17.954 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-19T10:32:17.969 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 96 M 2023-12-19T10:32:17.970 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 121 M 2023-12-19T10:32:17.971 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:32:18.723 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 12 MB/s | 5.6 MB 00:00 2023-12-19T10:32:19.315 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 19 MB/s | 20 MB 00:01 2023-12-19T10:32:20.332 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 18 MB/s | 38 MB 00:02 2023-12-19T10:32:20.674 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 17 MB/s | 32 MB 00:01 2023-12-19T10:32:20.675 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:32:20.675 INFO:teuthology.orchestra.run.smithi033.stdout:Total 36 MB/s | 96 MB 00:02 2023-12-19T10:32:21.567 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:32:22.011 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:32:22.011 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:32:23.374 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:32:23.375 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:32:26.190 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:32:26.861 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-19T10:32:27.007 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T10:32:29.910 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T10:32:30.093 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T10:32:35.048 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T10:32:35.169 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:32:40.158 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:33:10.725 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:33:10.770 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:33:11.218 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:33:11.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-19T10:33:11.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-19T10:33:11.219 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-19T10:33:11.684 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-19T10:33:11.684 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:33:11.685 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:33:12.198 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T10:33:12.344 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T10:33:12.345 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T10:33:12.345 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-19T10:33:12.345 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-19T10:33:12.345 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-19T10:33:12.345 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-19T10:33:12.345 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2023-12-19T10:33:12.345 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2023-12-19T10:33:12.469 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-19T10:33:12.469 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-19T10:33:12.469 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2023-12-19T10:33:12.506 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2023-12-19T10:33:12.506 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2023-12-19T10:33:12.506 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2023-12-19T10:33:12.507 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:33:12.508 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2023-12-19T10:33:12.509 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-19T10:33:12.509 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-391.el9.x86_64 2023-12-19T10:33:12.509 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-19T10:33:13.099 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2023-12-19T10:33:14.488 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-19T10:33:14.526 INFO:teuthology.orchestra.run.smithi033.stderr:done 2023-12-19T10:33:14.528 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-19T10:33:14.544 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.HtaZmFGcqP 2023-12-19T10:33:14.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HtaZmFGcqP 2023-12-19T10:33:14.616 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.HtaZmFGcqP 2023-12-19T10:33:14.760 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.HtaZmFGcqP is 6KB 2023-12-19T10:33:14.809 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.HtaZmFGcqP 2023-12-19T10:33:14.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2023-12-19T10:33:14.893 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-19T10:33:14.894 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-19T10:33:14.894 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-19T10:33:14.894 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-19T10:33:14.895 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-19T10:33:15.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2023-12-19T10:33:15.120 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2023-12-19T10:33:45.064 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-19T10:33:45.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:33:45.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T10:34:03.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-12-19T10:34:12.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:34:12.422 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T10:35:12.482 DEBUG:teuthology.orchestra.remote:timed out 2023-12-19T10:35:24.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:35:24.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T10:35:24.835 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-19T10:35:25.215 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:35:25.215 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-19T10:35:25.216 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2023-12-19T10:35:25.231 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-391.el9.x86_64 2023-12-19T10:35:25.231 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-19T10:35:25.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-19T10:35:25.231 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-19T10:35:26.232 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-19T10:35:26.232 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-19T10:35:26.320 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2023-12-19T10:35:26.353 DEBUG:teuthology.parallel:result is None 2023-12-19T10:35:26.354 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-19T10:35:26.361 INFO:teuthology.task.internal:Creating test directory... 2023-12-19T10:35:26.361 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-19T10:35:26.379 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-19T10:35:26.385 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-19T10:35:26.392 INFO:teuthology.task.internal:Creating archive directory... 2023-12-19T10:35:26.392 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-19T10:35:26.450 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-19T10:35:26.456 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-19T10:35:26.457 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-19T10:35:26.529 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T10:35:26.546 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-19T10:35:26.547 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-19T10:35:26.554 INFO:teuthology.task.internal:Configuring sudo... 2023-12-19T10:35:26.556 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-19T10:35:26.622 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-19T10:35:26.631 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-19T10:35:26.631 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-19T10:35:26.678 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T10:35:26.764 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-19T10:35:26.820 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T10:35:26.902 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-19T10:35:26.950 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:35:26.950 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-19T10:35:27.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2023-12-19T10:35:27.110 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T10:35:27.421 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-19T10:35:27.428 INFO:teuthology.task.internal:Starting timer... 2023-12-19T10:35:27.429 INFO:teuthology.run_tasks:Running task pcp... 2023-12-19T10:35:27.438 INFO:teuthology.run_tasks:Running task selinux... 2023-12-19T10:35:27.447 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2023-12-19T10:35:27.503 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2023-12-19T10:35:27.504 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-19T10:35:27.504 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2023-12-19T10:35:27.589 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2023-12-19T10:35:27.589 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi033.front.sepia.ceph.com': 'permissive'} 2023-12-19T10:35:27.589 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T10:35:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T10:35:27.622 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-19T10:35:27.622 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2023-12-19T10:35:27.699 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-19T10:35:27.709 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-19T10:35:27.709 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-19T10:35:27.725 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-19T10:35:27.726 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-19T10:41:08.778 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2023-12-19T10:41:08.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:41:08.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-19T10:41:08.877 DEBUG:teuthology.orchestra.run.smithi033:> true 2023-12-19T10:41:08.944 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-12-19T10:41:08.945 INFO:teuthology.run_tasks:Running task clock... 2023-12-19T10:41:08.955 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-19T10:41:08.955 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-19T10:41:08.956 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T10:41:09.020 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-19T10:41:09.040 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-19T10:41:09.077 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2023-12-19T10:41:09.093 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2023-12-19T10:41:09.113 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-19T10:41:09.133 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-19T10:41:09.180 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2023-12-19T10:41:09.183 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T10:41:09.183 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2023-12-19T10:41:09.183 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T10:41:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T10:41:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T10:41:09.184 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T10:41:09.185 INFO:teuthology.run_tasks:Running task install... 2023-12-19T10:41:09.194 DEBUG:teuthology.task.install:project ceph 2023-12-19T10:41:09.194 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'}} 2023-12-19T10:41:09.194 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'} 2023-12-19T10:41:09.194 INFO:teuthology.task.install:Using flavor: crimson 2023-12-19T10:41:09.200 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-19T10:41:09.200 INFO:teuthology.task.install:extra packages: [] 2023-12-19T10:41:09.200 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d', 'tag': None, 'wait_for_package': False} 2023-12-19T10:41:09.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=729869e4a72e72e756b33d15b6f2fbf113fa1a2d 2023-12-19T10:41:09.422 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-misc/729869e4a72e72e756b33d15b6f2fbf113fa1a2d/centos/9/flavors/crimson/ 2023-12-19T10:41:09.422 INFO:teuthology.task.install.rpm:Package version is 19.0.0-231.g729869e4 2023-12-19T10:41:09.793 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-misc/729869e4a72e72e756b33d15b6f2fbf113fa1a2d/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-misc/729869e4a72e72e756b33d15b6f2fbf113fa1a2d/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-misc/729869e4a72e72e756b33d15b6f2fbf113fa1a2d/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-19T10:41:09.793 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:41:09.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-19T10:41:09.828 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-19T10:41:09.828 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/729869e4a72e72e756b33d15b6f2fbf113fa1a2d/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-19T10:41:09.902 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-19T10:41:09.991 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-19T10:41:10.021 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2023-12-19T10:41:10.023 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2023-12-19T10:41:10.276 INFO:teuthology.orchestra.run.smithi033.stdout:73 files removed 2023-12-19T10:41:10.303 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw 2023-12-19T10:41:11.521 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 93 kB/s | 82 kB 00:00 2023-12-19T10:41:12.254 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 14 kB/s | 9.5 kB 00:00 2023-12-19T10:41:13.104 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2023-12-19T10:41:13.657 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2023-12-19T10:41:15.032 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-12-19T10:41:17.581 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 7.9 MB 00:02 2023-12-19T10:41:23.890 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 3.5 MB/s | 18 MB 00:05 2023-12-19T10:41:27.907 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-19T10:41:29.425 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2023-12-19T10:41:34.412 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 20 MB 00:04 2023-12-19T10:41:39.292 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-12-19T10:41:41.889 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:41:41.900 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-231.g729869e4.el9 ceph 8.3 M 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-231.g729869e4.el9 ceph 3.1 M 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-231.g729869e4.el9 ceph 3.0 M 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-231.g729869e4.el9 ceph 6.6 M 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-231.g729869e4.el9 ceph 17 M 2023-12-19T10:41:41.901 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-231.g729869e4.el9 ceph 24 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-231.g729869e4.el9 ceph 728 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-231.g729869e4.el9 ceph 469 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-231.g729869e4.el9 ceph 4.5 M 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse x86_64 2:19.0.0-231.g729869e4.el9 ceph 45 k 2023-12-19T10:41:41.902 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-231.g729869e4.el9 ceph 130 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-231.g729869e4.el9 ceph 162 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-231.g729869e4.el9 ceph 321 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-231.g729869e4.el9 ceph 298 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-231.g729869e4.el9 ceph 100 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout:Install 27 Packages 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 2 Packages 2023-12-19T10:41:41.903 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:41:41.904 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 54 M 2023-12-19T10:41:41.904 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:41:46.302 INFO:teuthology.orchestra.run.smithi033.stdout:(1/29): ceph-base-19.0.0-231.g729869e4.el9.x86_ 1.5 MB/s | 6.6 MB 00:04 2023-12-19T10:41:46.410 INFO:teuthology.orchestra.run.smithi033.stdout:(2/29): ceph-selinux-19.0.0-231.g729869e4.el9.x 223 kB/s | 24 kB 00:00 2023-12-19T10:41:46.897 INFO:teuthology.orchestra.run.smithi033.stdout:(3/29): libcephfs2-19.0.0-231.g729869e4.el9.x86 1.5 MB/s | 728 kB 00:00 2023-12-19T10:41:47.197 INFO:teuthology.orchestra.run.smithi033.stdout:(4/29): libradosstriper1-19.0.0-231.g729869e4.e 1.5 MB/s | 469 kB 00:00 2023-12-19T10:41:47.939 INFO:teuthology.orchestra.run.smithi033.stdout:(5/29): ceph-radosgw-19.0.0-231.g729869e4.el9.x 1.4 MB/s | 8.3 MB 00:05 2023-12-19T10:41:48.064 INFO:teuthology.orchestra.run.smithi033.stdout:(6/29): python3-ceph-argparse-19.0.0-231.g72986 361 kB/s | 45 kB 00:00 2023-12-19T10:41:48.173 INFO:teuthology.orchestra.run.smithi033.stdout:(7/29): python3-ceph-common-19.0.0-231.g729869e 1.2 MB/s | 130 kB 00:00 2023-12-19T10:41:48.365 INFO:teuthology.orchestra.run.smithi033.stdout:(8/29): python3-cephfs-19.0.0-231.g729869e4.el9 846 kB/s | 162 kB 00:00 2023-12-19T10:41:48.649 INFO:teuthology.orchestra.run.smithi033.stdout:(9/29): python3-rados-19.0.0-231.g729869e4.el9. 1.1 MB/s | 321 kB 00:00 2023-12-19T10:41:48.940 INFO:teuthology.orchestra.run.smithi033.stdout:(10/29): python3-rbd-19.0.0-231.g729869e4.el9.x 1.0 MB/s | 298 kB 00:00 2023-12-19T10:41:49.049 INFO:teuthology.orchestra.run.smithi033.stdout:(11/29): python3-rgw-19.0.0-231.g729869e4.el9.x 918 kB/s | 100 kB 00:00 2023-12-19T10:41:49.166 INFO:teuthology.orchestra.run.smithi033.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 908 kB/s | 106 kB 00:00 2023-12-19T10:41:49.208 INFO:teuthology.orchestra.run.smithi033.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-12-19T10:41:49.233 INFO:teuthology.orchestra.run.smithi033.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2023-12-19T10:41:49.283 INFO:teuthology.orchestra.run.smithi033.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-19T10:41:49.317 INFO:teuthology.orchestra.run.smithi033.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 8.3 MB/s | 282 kB 00:00 2023-12-19T10:41:49.343 INFO:teuthology.orchestra.run.smithi033.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2023-12-19T10:41:49.710 INFO:teuthology.orchestra.run.smithi033.stdout:(18/29): librgw2-19.0.0-231.g729869e4.el9.x86_6 1.8 MB/s | 4.5 MB 00:02 2023-12-19T10:41:49.960 INFO:teuthology.orchestra.run.smithi033.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 501 kB/s | 309 kB 00:00 2023-12-19T10:41:50.052 INFO:teuthology.orchestra.run.smithi033.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 284 kB/s | 26 kB 00:00 2023-12-19T10:41:50.152 INFO:teuthology.orchestra.run.smithi033.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2023-12-19T10:41:50.260 INFO:teuthology.orchestra.run.smithi033.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2023-12-19T10:41:50.527 INFO:teuthology.orchestra.run.smithi033.stdout:(23/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2023-12-19T10:41:50.619 INFO:teuthology.orchestra.run.smithi033.stdout:(24/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2023-12-19T10:41:50.653 INFO:teuthology.orchestra.run.smithi033.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-12-19T10:41:50.829 INFO:teuthology.orchestra.run.smithi033.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2023-12-19T10:41:53.637 INFO:teuthology.orchestra.run.smithi033.stdout:(27/29): ceph-common-19.0.0-231.g729869e4.el9.x 1.5 MB/s | 17 MB 00:11 2023-12-19T10:41:53.687 INFO:teuthology.orchestra.run.smithi033.stdout:(28/29): librados2-19.0.0-231.g729869e4.el9.x86 1.0 MB/s | 3.1 MB 00:03 2023-12-19T10:41:53.746 INFO:teuthology.orchestra.run.smithi033.stdout:(29/29): librbd1-19.0.0-231.g729869e4.el9.x86_6 1.0 MB/s | 3.0 MB 00:02 2023-12-19T10:41:53.749 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:41:53.749 INFO:teuthology.orchestra.run.smithi033.stdout:Total 4.5 MB/s | 54 MB 00:11 2023-12-19T10:41:53.750 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:41:53.791 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:41:53.791 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:41:54.265 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:41:54.266 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:41:55.667 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:41:56.050 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-19T10:41:56.115 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-231.g729869e4.el9.x86_64 2/31 2023-12-19T10:41:56.207 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-231.g729869e4.el9.x86_64 2/31 2023-12-19T10:41:56.298 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-231.g729869e4.el9.x86_64 3/31 2023-12-19T10:41:56.324 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-231.g729869e4.el9.x86_64 4/31 2023-12-19T10:41:56.427 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-231.g729869e4.el9.x86_64 4/31 2023-12-19T10:41:56.461 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-12-19T10:41:56.487 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-12-19T10:41:56.525 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-12-19T10:41:56.570 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-231.g729869e4.el9.x 8/31 2023-12-19T10:41:56.638 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-231.g729869e4.el9.x86_64 9/31 2023-12-19T10:41:56.661 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-231.g729869e4.el9.x86_64 10/31 2023-12-19T10:41:56.753 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-231.g729869e4.el9.x86_64 10/31 2023-12-19T10:41:56.789 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-12-19T10:41:56.899 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-19T10:41:56.932 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2023-12-19T10:41:56.973 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-19T10:41:57.004 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-19T10:41:57.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2023-12-19T10:41:57.050 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-19T10:41:57.380 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-19T10:41:57.406 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-231.g729869e4.el9.x86_64 19/31 2023-12-19T10:41:57.480 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-231.g729869e4.el9.x86_64 19/31 2023-12-19T10:41:57.511 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-231.g729869e4.el9.x86_64 20/31 2023-12-19T10:41:57.547 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-19T10:41:57.996 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2023-12-19T10:41:58.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-231.g729869e4.el9.x86_64 23/31 2023-12-19T10:41:58.080 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-231.g729869e4.el9.x86_64 23/31 2023-12-19T10:41:58.151 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-231.g729869e4.el9.x86_64 24/31 2023-12-19T10:41:58.178 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-231.g729869e4.el9.x86 25/31 2023-12-19T10:42:00.324 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 26/31 2023-12-19T10:42:00.358 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 26/31 2023-12-19T10:42:01.139 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 26/31 2023-12-19T10:42:01.227 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-231.g729869e4.el9.x86_64 27/31 2023-12-19T10:42:01.305 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-231.g729869e4.el9.x86_64 27/31 2023-12-19T10:42:01.305 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-19T10:42:01.305 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-19T10:42:01.305 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:01.337 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-231.g729869e4.el9.x86_64 28/31 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-231.g729869e4.el9.x86_64 28/31 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2023-12-19T10:42:11.341 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2023-12-19T10:42:11.342 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:12.201 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-231.g729869e4.el9.x86_64 29/31 2023-12-19T10:42:12.226 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-231.g729869e4.el9.x86_64 29/31 2023-12-19T10:42:12.226 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:42:12.226 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-19T10:42:12.226 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T10:42:12.227 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-19T10:42:12.227 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:12.228 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-19T10:42:12.311 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-19T10:42:12.311 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-19T10:42:13.488 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-231.g729869e4.el9.x86_64 1/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 2/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-231.g729869e4.el9.x86_64 3/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-231.g729869e4.el9.x86_64 4/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-231.g729869e4.el9.x86_64 5/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-231.g729869e4.el9.x86_64 6/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-231.g729869e4.el9.x86_64 7/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-231.g729869e4.el9.x 8/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-231.g729869e4.el9.x86 9/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-231.g729869e4.el9.x86_64 10/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-231.g729869e4.el9.x86_64 11/31 2023-12-19T10:42:13.489 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-231.g729869e4.el9.x86_64 12/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-231.g729869e4.el9.x86_64 13/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-19T10:42:13.490 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-19T10:42:13.491 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-231.g729869e4.el9.x86_64 28/31 2023-12-19T10:42:13.491 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-19T10:42:13.491 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-231.g729869e4.el9.x86_64 30/31 2023-12-19T10:42:14.328 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-19T10:42:14.329 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.330 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:14.331 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:42:14.597 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test 2023-12-19T10:42:15.175 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:00:36 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:42:15.324 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:42:15.325 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:42:15.325 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:42:15.325 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:42:15.325 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-231.g729869e4.el9 ceph 118 M 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-231.g729869e4.el9 ceph 158 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:15.326 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout:Install 10 Packages 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 120 M 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.1 G 2023-12-19T10:42:15.327 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:42:15.492 INFO:teuthology.orchestra.run.smithi033.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 186 kB 00:00 2023-12-19T10:42:15.592 INFO:teuthology.orchestra.run.smithi033.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 7.8 MB/s | 418 kB 00:00 2023-12-19T10:42:15.618 INFO:teuthology.orchestra.run.smithi033.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 7.2 MB/s | 186 kB 00:00 2023-12-19T10:42:15.651 INFO:teuthology.orchestra.run.smithi033.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.6 MB/s | 217 kB 00:00 2023-12-19T10:42:15.684 INFO:teuthology.orchestra.run.smithi033.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 8.8 MB/s | 300 kB 00:00 2023-12-19T10:42:15.718 INFO:teuthology.orchestra.run.smithi033.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-19T10:42:15.968 INFO:teuthology.orchestra.run.smithi033.stdout:(7/10): cryptopp-8.6.0-1.el9.x86_64.rpm 5.0 MB/s | 1.3 MB 00:00 2023-12-19T10:42:16.010 INFO:teuthology.orchestra.run.smithi033.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 2.9 MB/s | 122 kB 00:00 2023-12-19T10:42:16.177 INFO:teuthology.orchestra.run.smithi033.stdout:(9/10): libcephsqlite-19.0.0-231.g729869e4.el9. 199 kB/s | 158 kB 00:00 2023-12-19T10:43:20.444 INFO:teuthology.orchestra.run.smithi033.stdout:(10/10): ceph-test-19.0.0-231.g729869e4.el9.x86 1.8 MB/s | 118 MB 01:05 2023-12-19T10:43:20.446 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:43:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.8 MB/s | 120 MB 01:05 2023-12-19T10:43:20.447 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:43:20.470 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:43:20.470 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:43:21.327 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:43:21.328 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:43:22.498 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:43:22.589 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-12-19T10:43:22.635 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-12-19T10:43:22.685 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-12-19T10:43:22.730 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-12-19T10:43:22.780 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-12-19T10:43:22.810 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-12-19T10:43:22.845 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-12-19T10:43:22.880 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-12-19T10:43:22.907 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-231.g729869e4.el9.x86_64 9/10 2023-12-19T10:43:40.074 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-231.g729869e4.el9.x86_64 9/10 2023-12-19T10:43:40.149 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-231.g729869e4.el9.x86_64 10/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-2:19.0.0-231.g729869e4.el9.x86_64 10/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-231.g729869e4.el9.x86_64 1/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-231.g729869e4.el9.x86_64 2/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-12-19T10:43:48.262 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-12-19T10:43:48.263 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-12-19T10:43:48.263 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-12-19T10:43:48.263 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-12-19T10:43:48.263 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-12-19T10:43:48.686 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-12-19T10:43:48.686 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:43:48.686 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:43:48.686 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:43:48.687 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:43:49.002 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph 2023-12-19T10:43:49.582 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:02:10 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:43:49.736 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:43:49.738 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2023-12-19T10:43:49.738 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:43:49.738 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2023-12-19T10:43:49.738 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:43:49.738 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-231.g729869e4.el9 ceph 6.5 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-231.g729869e4.el9 ceph 2.0 M 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-231.g729869e4.el9 ceph 1.4 M 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 243 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-231.g729869e4.el9 ceph 8.9 M 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-231.g729869e4.el9 ceph 117 M 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-19T10:43:49.739 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-19T10:43:49.740 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout:Install 35 Packages 2023-12-19T10:43:49.741 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:43:49.742 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 134 M 2023-12-19T10:43:49.742 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.4 G 2023-12-19T10:43:49.742 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:43:50.196 INFO:teuthology.orchestra.run.smithi033.stdout:(1/35): ceph-19.0.0-231.g729869e4.el9.x86_64.rp 18 kB/s | 6.5 kB 00:00 2023-12-19T10:43:51.421 INFO:teuthology.orchestra.run.smithi033.stdout:(2/35): ceph-mgr-19.0.0-231.g729869e4.el9.x86_6 930 kB/s | 1.4 MB 00:01 2023-12-19T10:43:51.513 INFO:teuthology.orchestra.run.smithi033.stdout:(3/35): ceph-mds-19.0.0-231.g729869e4.el9.x86_6 1.2 MB/s | 2.0 MB 00:01 2023-12-19T10:43:51.697 INFO:teuthology.orchestra.run.smithi033.stdout:(4/35): ceph-mgr-modules-core-19.0.0-231.g72986 1.3 MB/s | 243 kB 00:00 2023-12-19T10:43:51.806 INFO:teuthology.orchestra.run.smithi033.stdout:(5/35): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2023-12-19T10:43:51.847 INFO:teuthology.orchestra.run.smithi033.stdout:(6/35): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-19T10:43:51.872 INFO:teuthology.orchestra.run.smithi033.stdout:(7/35): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-12-19T10:43:51.898 INFO:teuthology.orchestra.run.smithi033.stdout:(8/35): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2023-12-19T10:43:51.923 INFO:teuthology.orchestra.run.smithi033.stdout:(9/35): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-12-19T10:43:51.965 INFO:teuthology.orchestra.run.smithi033.stdout:(10/35): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-12-19T10:43:52.032 INFO:teuthology.orchestra.run.smithi033.stdout:(11/35): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2023-12-19T10:43:52.066 INFO:teuthology.orchestra.run.smithi033.stdout:(12/35): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-12-19T10:43:52.128 INFO:teuthology.orchestra.run.smithi033.stdout:(13/35): python3-pycparser-2.20-6.el9.noarch.rp 1.9 MB/s | 124 kB 00:00 2023-12-19T10:43:52.158 INFO:teuthology.orchestra.run.smithi033.stdout:(14/35): python3-requests-2.25.1-7.el9.noarch.r 3.8 MB/s | 113 kB 00:00 2023-12-19T10:43:52.192 INFO:teuthology.orchestra.run.smithi033.stdout:(15/35): python3-urllib3-1.26.5-4.el9.noarch.rp 5.4 MB/s | 187 kB 00:00 2023-12-19T10:43:52.925 INFO:teuthology.orchestra.run.smithi033.stdout:(16/35): luarocks-3.9.2-1.el9.noarch.rpm 206 kB/s | 151 kB 00:00 2023-12-19T10:43:53.042 INFO:teuthology.orchestra.run.smithi033.stdout:(17/35): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2023-12-19T10:43:53.318 INFO:teuthology.orchestra.run.smithi033.stdout:(18/35): python3-cheroot-8.6.0-4.el9.noarch.rpm 625 kB/s | 172 kB 00:00 2023-12-19T10:43:53.826 INFO:teuthology.orchestra.run.smithi033.stdout:(19/35): python3-cherrypy-18.6.1-2.el9.noarch.r 705 kB/s | 358 kB 00:00 2023-12-19T10:43:53.926 INFO:teuthology.orchestra.run.smithi033.stdout:(20/35): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2023-12-19T10:43:54.035 INFO:teuthology.orchestra.run.smithi033.stdout:(21/35): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2023-12-19T10:43:54.152 INFO:teuthology.orchestra.run.smithi033.stdout:(22/35): python3-jaraco-collections-3.0.0-8.el9 199 kB/s | 23 kB 00:00 2023-12-19T10:43:54.252 INFO:teuthology.orchestra.run.smithi033.stdout:(23/35): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2023-12-19T10:43:54.361 INFO:teuthology.orchestra.run.smithi033.stdout:(24/35): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-12-19T10:43:54.469 INFO:teuthology.orchestra.run.smithi033.stdout:(25/35): python3-logutils-0.3.5-21.el9.noarch.r 428 kB/s | 46 kB 00:00 2023-12-19T10:43:54.578 INFO:teuthology.orchestra.run.smithi033.stdout:(26/35): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2023-12-19T10:43:55.004 INFO:teuthology.orchestra.run.smithi033.stdout:(27/35): python3-pecan-1.4.2-3.el9.noarch.rpm 641 kB/s | 272 kB 00:00 2023-12-19T10:43:55.112 INFO:teuthology.orchestra.run.smithi033.stdout:(28/35): python3-portend-3.1.0-2.el9.noarch.rpm 152 kB/s | 16 kB 00:00 2023-12-19T10:43:55.296 INFO:teuthology.orchestra.run.smithi033.stdout:(29/35): python3-pyOpenSSL-21.0.0-1.el9.noarch. 491 kB/s | 90 kB 00:00 2023-12-19T10:43:55.404 INFO:teuthology.orchestra.run.smithi033.stdout:(30/35): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2023-12-19T10:43:55.755 INFO:teuthology.orchestra.run.smithi033.stdout:(31/35): python3-webob-1.8.7-6.el9.noarch.rpm 656 kB/s | 230 kB 00:00 2023-12-19T10:43:56.388 INFO:teuthology.orchestra.run.smithi033.stdout:(32/35): python3-werkzeug-2.0.3-3.el9.1.noarch. 674 kB/s | 427 kB 00:00 2023-12-19T10:43:56.513 INFO:teuthology.orchestra.run.smithi033.stdout:(33/35): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2023-12-19T10:43:59.338 INFO:teuthology.orchestra.run.smithi033.stdout:(34/35): ceph-mon-19.0.0-231.g729869e4.el9.x86_ 992 kB/s | 8.9 MB 00:09 2023-12-19T10:44:28.693 INFO:teuthology.orchestra.run.smithi033.stdout:(35/35): ceph-osd-19.0.0-231.g729869e4.el9.x86_ 3.1 MB/s | 117 MB 00:37 2023-12-19T10:44:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:44:28.697 INFO:teuthology.orchestra.run.smithi033.stdout:Total 3.4 MB/s | 134 MB 00:38 2023-12-19T10:44:28.698 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:44:28.740 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:44:28.741 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:44:29.833 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:44:29.834 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:44:31.760 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:44:31.820 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/35 2023-12-19T10:44:31.853 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/35 2023-12-19T10:44:31.886 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/35 2023-12-19T10:44:31.981 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/35 2023-12-19T10:44:32.067 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/35 2023-12-19T10:44:32.114 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/35 2023-12-19T10:44:32.188 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/35 2023-12-19T10:44:32.239 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/35 2023-12-19T10:44:32.278 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/35 2023-12-19T10:44:32.336 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/35 2023-12-19T10:44:32.399 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/35 2023-12-19T10:44:32.600 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-19T10:44:32.656 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-19T10:44:32.693 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/35 2023-12-19T10:44:32.730 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/35 2023-12-19T10:44:32.759 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/35 2023-12-19T10:44:32.839 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/35 2023-12-19T10:44:32.878 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/35 2023-12-19T10:44:32.911 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/35 2023-12-19T10:44:32.952 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/35 2023-12-19T10:44:32.996 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/35 2023-12-19T10:44:33.046 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/35 2023-12-19T10:44:33.140 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/35 2023-12-19T10:44:33.182 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/35 2023-12-19T10:44:33.261 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/35 2023-12-19T10:44:33.402 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/35 2023-12-19T10:44:33.534 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/35 2023-12-19T10:44:33.726 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-231.g729869e4.el9.n 28/35 2023-12-19T10:44:33.757 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-231.g729869e4.el9.x86_64 29/35 2023-12-19T10:44:33.783 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-231.g729869e4.el9.x86_64 29/35 2023-12-19T10:44:33.783 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:44:33.783 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-19T10:44:33.784 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T10:44:33.784 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-19T10:44:33.784 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:44:33.875 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 30/35 2023-12-19T10:44:53.299 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 31/35 2023-12-19T10:44:53.333 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-231.g729869e4.el9.x86_64 32/35 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-231.g729869e4.el9.x86_64 32/35 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-19T10:44:53.360 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:44:54.292 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-231.g729869e4.el9.x86_64 33/35 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-231.g729869e4.el9.x86_64 33/35 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-19T10:44:54.317 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:44:54.525 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-231.g729869e4.el9.x86_64 34/35 2023-12-19T10:44:54.549 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-231.g729869e4.el9.x86_64 34/35 2023-12-19T10:44:54.549 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:44:54.549 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-19T10:44:54.550 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T10:44:54.550 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-19T10:44:54.550 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:44:54.617 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-231.g729869e4.el9.x86_64 35/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-2:19.0.0-231.g729869e4.el9.x86_64 35/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-231.g729869e4.el9.x86_64 1/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-231.g729869e4.el9.x86_64 2/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-231.g729869e4.el9.x86_64 3/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-231.g729869e4.el9.x86_64 4/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-231.g729869e4.el9.x86_64 5/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-231.g729869e4.el9.n 6/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/35 2023-12-19T10:45:04.347 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 18/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 19/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 20/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 21/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 22/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 23/35 2023-12-19T10:45:04.348 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 24/35 2023-12-19T10:45:04.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 25/35 2023-12-19T10:45:04.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 26/35 2023-12-19T10:45:04.349 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 27/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 28/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 29/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 30/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 31/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 32/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 33/35 2023-12-19T10:45:04.350 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 34/35 2023-12-19T10:45:05.353 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 35/35 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-19T10:45:05.354 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2023-12-19T10:45:05.355 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:05.356 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:05.604 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base 2023-12-19T10:45:06.186 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:27 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:06.285 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:06.337 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:06.338 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:06.338 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:06.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install cephadm 2023-12-19T10:45:07.002 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:28 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:07.154 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 805 k 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:07.155 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 805 k 2023-12-19T10:45:07.156 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 823 k 2023-12-19T10:45:07.156 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:07.907 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm-19.0.0-231.g729869e4.el9.noarch.rpm 1.0 MB/s | 805 kB 00:00 2023-12-19T10:45:07.907 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:07.908 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.0 MB/s | 805 kB 00:00 2023-12-19T10:45:07.908 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:07.914 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:07.915 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:07.921 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:07.922 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:07.996 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:08.449 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:08.549 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:08.983 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:09.335 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:09.335 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:09.335 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:09.335 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:09.336 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:09.336 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:09.545 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache 2023-12-19T10:45:10.131 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:31 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:10.279 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:10.280 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:10.280 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2023-12-19T10:45:10.280 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:10.280 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-231.g729869e4.el9 ceph 143 k 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 143 k 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 438 k 2023-12-19T10:45:10.281 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:10.504 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-19.0.0-231.g729869e 645 kB/s | 143 kB 00:00 2023-12-19T10:45:10.504 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:10.505 INFO:teuthology.orchestra.run.smithi033.stdout:Total 641 kB/s | 143 kB 00:00 2023-12-19T10:45:10.505 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:10.513 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:10.513 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:10.568 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:10.568 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:10.722 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:10.788 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-231.g729869e4.e 1/1 2023-12-19T10:45:10.814 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-231.g729869e4.e 1/1 2023-12-19T10:45:10.814 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:45:10.814 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-19T10:45:10.814 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-231.g729869e4.e 1/1 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:11.807 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:12.176 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr 2023-12-19T10:45:12.769 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:33 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:12.866 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-mgr-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:12.918 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:12.919 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:13.001 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-dashboard 2023-12-19T10:45:13.584 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:34 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:13.734 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 1.8 M 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 28 k 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 15 k 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-19T10:45:13.735 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout:Install 7 Packages 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.1 M 2023-12-19T10:45:13.736 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 8.7 M 2023-12-19T10:45:13.737 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:14.166 INFO:teuthology.orchestra.run.smithi033.stdout:(1/7): ceph-prometheus-alerts-19.0.0-231.g72986 60 kB/s | 15 kB 00:00 2023-12-19T10:45:14.191 INFO:teuthology.orchestra.run.smithi033.stdout:(2/7): ceph-grafana-dashboards-19.0.0-231.g7298 102 kB/s | 28 kB 00:00 2023-12-19T10:45:14.316 INFO:teuthology.orchestra.run.smithi033.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 60 kB/s | 9.0 kB 00:00 2023-12-19T10:45:14.358 INFO:teuthology.orchestra.run.smithi033.stdout:(4/7): ceph-mgr-dashboard-19.0.0-231.g729869e4. 4.0 MB/s | 1.8 MB 00:00 2023-12-19T10:45:14.383 INFO:teuthology.orchestra.run.smithi033.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 461 kB/s | 31 kB 00:00 2023-12-19T10:45:14.433 INFO:teuthology.orchestra.run.smithi033.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 169 kB/s | 41 kB 00:00 2023-12-19T10:45:14.559 INFO:teuthology.orchestra.run.smithi033.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 940 kB/s | 188 kB 00:00 2023-12-19T10:45:14.563 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:14.563 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.5 MB/s | 2.1 MB 00:00 2023-12-19T10:45:14.563 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:14.571 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:14.571 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:14.640 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:14.640 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:14.952 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:15.010 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-19T10:45:15.060 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-19T10:45:15.108 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-19T10:45:15.139 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-19T10:45:15.169 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-231.g729869e4.el9.no 5/7 2023-12-19T10:45:15.419 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-231.g729869e4.el9.n 6/7 2023-12-19T10:45:15.446 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-231.g729869e4.el9.noarch 7/7 2023-12-19T10:45:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-231.g729869e4.el9.noarch 7/7 2023-12-19T10:45:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-231.g729869e4.el9.n 1/7 2023-12-19T10:45:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-231.g729869e4.el9.noarch 2/7 2023-12-19T10:45:15.982 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-231.g729869e4.el9.no 3/7 2023-12-19T10:45:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-19T10:45:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-19T10:45:15.983 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-19T10:45:16.303 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-19T10:45:16.303 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:16.303 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:16.304 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:16.553 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-19T10:45:17.133 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:38 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:17.285 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout:=========================================================================================== 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout:=========================================================================================== 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 7.4 M 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:45:17.286 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout:=========================================================================================== 2023-12-19T10:45:17.287 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2023-12-19T10:45:17.288 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:17.288 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 37 M 2023-12-19T10:45:17.288 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 209 M 2023-12-19T10:45:17.288 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:17.557 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 257 kB/s | 32 kB 00:00 2023-12-19T10:45:17.591 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 538 kB/s | 17 kB 00:00 2023-12-19T10:45:17.652 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2023-12-19T10:45:17.675 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 419 kB/s | 34 kB 00:00 2023-12-19T10:45:17.792 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 33 MB/s | 4.7 MB 00:00 2023-12-19T10:45:17.859 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-12-19T10:45:17.884 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2023-12-19T10:45:17.942 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-12-19T10:45:18.126 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-12-19T10:45:18.151 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 946 kB/s | 195 kB 00:00 2023-12-19T10:45:18.285 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2023-12-19T10:45:18.377 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 7.8 MB/s | 7.4 MB 00:00 2023-12-19T10:45:18.379 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:18.379 INFO:teuthology.orchestra.run.smithi033.stdout:Total 34 MB/s | 37 MB 00:01 2023-12-19T10:45:18.392 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:18.417 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:18.417 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:18.833 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:18.834 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:19.557 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:19.633 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-19T10:45:19.681 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-19T10:45:19.780 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-19T10:45:19.818 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-19T10:45:20.002 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T10:45:20.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T10:45:20.100 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-19T10:45:20.803 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-19T10:45:20.908 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-19T10:45:22.204 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-19T10:45:23.280 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-19T10:45:23.298 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-231.g729869 12/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-231.g729869 12/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-231.g729869 1/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-19T10:45:26.009 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-19T10:45:26.010 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-19T10:45:26.010 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-19T10:45:26.010 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-19T10:45:26.469 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-19T10:45:26.470 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-19T10:45:26.471 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-19T10:45:26.471 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:26.471 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:26.846 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-rook 2023-12-19T10:45:27.432 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:48 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:27.585 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:27.586 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:27.586 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:27.586 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:27.586 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 50 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-19T10:45:27.587 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:27.588 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.0 M 2023-12-19T10:45:27.589 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2023-12-19T10:45:27.589 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:27.883 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 153 kB/s | 18 kB 00:00 2023-12-19T10:45:27.902 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 181 kB/s | 24 kB 00:00 2023-12-19T10:45:27.942 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.9 MB/s | 174 kB 00:00 2023-12-19T10:45:27.967 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-19T10:45:27.992 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-19T10:45:28.017 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2023-12-19T10:45:28.034 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): ceph-mgr-rook-19.0.0-231.g729869e4.el9. 186 kB/s | 50 kB 00:00 2023-12-19T10:45:28.093 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 321 kB/s | 32 kB 00:00 2023-12-19T10:45:28.152 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2023-12-19T10:45:28.177 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 1.4 MB/s | 197 kB 00:00 2023-12-19T10:45:28.227 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2023-12-19T10:45:28.252 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2023-12-19T10:45:28.378 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2023-12-19T10:45:28.379 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:28.380 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2023-12-19T10:45:28.380 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:28.394 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:28.394 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:28.502 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:28.502 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:29.186 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:29.374 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-19T10:45:29.439 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-19T10:45:29.503 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-19T10:45:29.547 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-19T10:45:29.588 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-19T10:45:29.685 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-19T10:45:29.795 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-19T10:45:29.858 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-19T10:45:30.619 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-19T10:45:30.670 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-19T10:45:30.703 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-19T10:45:30.752 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-19T10:45:30.781 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-231.g729869e4.el9.noarch 13/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-231.g729869e4.el9.noarch 13/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-231.g729869e4.el9.noarch 1/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-19T10:45:31.572 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-19T10:45:31.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-19T10:45:31.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-19T10:45:31.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-19T10:45:31.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-19T10:45:31.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-19T10:45:31.959 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-19T10:45:31.959 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-19T10:45:31.960 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:31.961 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:32.228 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-cephadm 2023-12-19T10:45:32.813 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:53 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:32.966 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 133 k 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-19T10:45:32.967 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================= 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.8 M 2023-12-19T10:45:32.968 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 33 M 2023-12-19T10:45:32.969 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:33.312 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-19T10:45:33.429 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2023-12-19T10:45:33.504 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): ceph-mgr-cephadm-19.0.0-231.g729869e4.el 375 kB/s | 133 kB 00:00 2023-12-19T10:45:33.596 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 346 kB/s | 58 kB 00:00 2023-12-19T10:45:33.663 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 321 kB/s | 51 kB 00:00 2023-12-19T10:45:33.688 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2023-12-19T10:45:33.690 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:33.690 INFO:teuthology.orchestra.run.smithi033.stdout:Total 9.4 MB/s | 6.8 MB 00:00 2023-12-19T10:45:33.690 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:33.701 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:33.701 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:33.798 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:33.798 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:34.115 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:34.294 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-19T10:45:34.358 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-19T10:45:34.917 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-19T10:45:34.995 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-19T10:45:35.064 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-19T10:45:35.094 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-231.g729869e4.el9.noarch 6/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-231.g729869e4.el9.noarch 6/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-231.g729869e4.el9.noarch 1/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-19T10:45:35.899 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-19T10:45:36.206 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:36.207 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:36.490 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse 2023-12-19T10:45:37.083 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:58 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:37.234 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:37.234 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:37.234 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-231.g729869e4.el9 ceph 833 k 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2023-12-19T10:45:37.235 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:37.236 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 911 k 2023-12-19T10:45:37.236 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 3.0 M 2023-12-19T10:45:37.236 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:37.358 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 647 kB/s | 79 kB 00:00 2023-12-19T10:45:37.566 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-fuse-19.0.0-231.g729869e4.el9.x86_6 2.5 MB/s | 833 kB 00:00 2023-12-19T10:45:37.567 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:37.568 INFO:teuthology.orchestra.run.smithi033.stdout:Total 2.7 MB/s | 911 kB 00:00 2023-12-19T10:45:37.568 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:37.589 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:37.589 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:37.669 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:37.669 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:37.866 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:38.003 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-19T10:45:38.076 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-231.g729869e4.el9.x86_64 2/2 2023-12-19T10:45:38.802 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-2:19.0.0-231.g729869e4.el9.x86_64 2/2 2023-12-19T10:45:38.803 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-231.g729869e4.el9.x86_64 1/2 2023-12-19T10:45:39.114 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-19T10:45:39.114 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:39.114 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:39.115 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-231.g729869e4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-19T10:45:39.115 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:39.115 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:39.397 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-volume 2023-12-19T10:45:39.988 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:00 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:40.141 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:40.141 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-231.g729869e4.el9 ceph-noarch 262 k 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 262 k 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.2 M 2023-12-19T10:45:40.142 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:40.537 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-volume-19.0.0-231.g729869e4.el9.noarch.rpm 666 kB/s | 262 kB 00:00 2023-12-19T10:45:40.537 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:40.538 INFO:teuthology.orchestra.run.smithi033.stdout:Total 663 kB/s | 262 kB 00:00 2023-12-19T10:45:40.538 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:40.546 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:40.546 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:40.586 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:40.589 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:40.814 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:40.897 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:40.914 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:40.914 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:45:40.914 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-19T10:45:40.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-231.g729869e4.el9.noarch 1/1 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-231.g729869e4.el9.noarch 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:41.946 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:42.280 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel 2023-12-19T10:45:42.875 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:03 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:43.026 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:43.032 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:43.032 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2023-12-19T10:45:43.032 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:43.032 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:43.032 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-231.g729869e4.el9 ceph 127 k 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 127 k 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 456 k 2023-12-19T10:45:43.033 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-19.0.0-231.g729869e4.el9.x86_64. 549 kB/s | 127 kB 00:00 2023-12-19T10:45:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:Total 545 kB/s | 127 kB 00:00 2023-12-19T10:45:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:43.265 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:43.265 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:43.325 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:43.326 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:43.479 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:43.584 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:44.076 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:44.370 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:44.370 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:44.371 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:44.371 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:44.371 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:44.371 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:44.637 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2 2023-12-19T10:45:45.225 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:06 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:45.324 INFO:teuthology.orchestra.run.smithi033.stdout:Package libcephfs2-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:45.379 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:45.381 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:45.381 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:45.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs-devel 2023-12-19T10:45:46.050 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:07 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:46.205 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-231.g729869e4.el9 ceph 31 k 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 31 k 2023-12-19T10:45:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 140 k 2023-12-19T10:45:46.207 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:46.503 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs-devel-19.0.0-231.g729869e4.el9.x86_64 105 kB/s | 31 kB 00:00 2023-12-19T10:45:46.503 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:46.504 INFO:teuthology.orchestra.run.smithi033.stdout:Total 104 kB/s | 31 kB 00:00 2023-12-19T10:45:46.504 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:46.507 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:46.507 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:46.537 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:46.538 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:46.633 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:46.770 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:47.158 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:47.439 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:47.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2 2023-12-19T10:45:48.244 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:09 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:48.341 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:48.395 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:48.396 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:48.397 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:48.477 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1 2023-12-19T10:45:49.067 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:10 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:49.165 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:49.220 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:49.221 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:49.221 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:49.306 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados 2023-12-19T10:45:49.895 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:10 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:49.992 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rados-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:50.045 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:50.046 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:50.046 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:50.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw 2023-12-19T10:45:50.713 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:11 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:50.813 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rgw-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:50.868 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:50.869 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:50.870 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:50.955 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs 2023-12-19T10:45:51.541 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:12 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:51.640 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-cephfs-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:51.694 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:51.695 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:51.695 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:51.776 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd 2023-12-19T10:45:52.362 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:13 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:52.460 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rbd-2:19.0.0-231.g729869e4.el9.x86_64 is already installed. 2023-12-19T10:45:52.514 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:52.515 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2023-12-19T10:45:52.516 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:52.594 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse 2023-12-19T10:45:53.185 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:14 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-231.g729869e4.el9 ceph 86 k 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:53.338 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:53.339 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:53.339 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:53.339 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 86 k 2023-12-19T10:45:53.339 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 256 k 2023-12-19T10:45:53.339 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:53.556 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-19.0.0-231.g729869e4.el9.x86_64.rpm 399 kB/s | 86 kB 00:00 2023-12-19T10:45:53.556 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:53.557 INFO:teuthology.orchestra.run.smithi033.stdout:Total 397 kB/s | 86 kB 00:00 2023-12-19T10:45:53.557 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:53.564 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:53.564 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:53.622 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:53.622 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:53.774 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:53.874 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:54.259 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:54.553 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:54.553 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:54.553 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:54.553 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:54.554 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:54.554 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:54.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror 2023-12-19T10:45:55.398 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:16 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:55.550 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-231.g729869e4.el9 ceph 3.0 M 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.0 M 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 13 M 2023-12-19T10:45:55.551 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:56.143 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-19.0.0-231.g729869e4.el9.x86_64.rpm 5.1 MB/s | 3.0 MB 00:00 2023-12-19T10:45:56.143 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:56.143 INFO:teuthology.orchestra.run.smithi033.stdout:Total 5.0 MB/s | 3.0 MB 00:00 2023-12-19T10:45:56.144 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:56.153 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:56.153 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:56.230 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:56.230 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:56.672 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:56.743 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-19T10:45:56.768 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:57.936 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:45:58.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd 2023-12-19T10:45:58.750 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:19 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:45:58.900 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-231.g729869e4.el9 ceph 169 k 2023-12-19T10:45:58.901 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 169 k 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 537 k 2023-12-19T10:45:58.902 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:45:59.120 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-19.0.0-231.g729869e4.el9.x86_64.rpm 777 kB/s | 169 kB 00:00 2023-12-19T10:45:59.120 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:45:59.120 INFO:teuthology.orchestra.run.smithi033.stdout:Total 773 kB/s | 169 kB 00:00 2023-12-19T10:45:59.121 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:45:59.129 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:45:59.129 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:45:59.189 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:45:59.189 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:45:59.340 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:45:59.455 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:45:59.917 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-231.g729869e4.el9.x86_64 1/1 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:00.193 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:46:00.450 DEBUG:teuthology.parallel:result is None 2023-12-19T10:46:00.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=729869e4a72e72e756b33d15b6f2fbf113fa1a2d 2023-12-19T10:46:00.688 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-19T10:46:00.710 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-231.g729869e4.el9 2023-12-19T10:46:00.711 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-231.g729869e4.el9 2023-12-19T10:46:00.711 INFO:teuthology.task.install:The correct ceph version 19.0.0-231.g729869e4 is installed. 2023-12-19T10:46:00.713 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-19T10:46:00.714 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:00.714 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-19T10:46:00.783 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-19T10:46:00.783 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:00.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2023-12-19T10:46:00.854 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-19T10:46:00.925 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-19T10:46:00.925 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:00.925 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-19T10:46:01.000 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-19T10:46:01.071 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-19T10:46:01.071 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:01.071 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2023-12-19T10:46:01.142 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-19T10:46:01.213 INFO:teuthology.run_tasks:Running task ceph... 2023-12-19T10:46:01.288 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-19T10:46:01.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2023-12-19T10:46:01.318 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-19T10:46:01.318 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-19T10:46:01.389 INFO:tasks.ceph:Creating extra log directories... 2023-12-19T10:46:01.389 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-19T10:46:01.467 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-19T10:46:01.467 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-19T10:46:01.468 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733', 'branch': 'wip-matanb-crimson-testing-misc', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7496733', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-misc'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'}}, 'workunit': {'branch': 'wip-matanb-crimson-testing-misc', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6360, 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-testing-misc', 'suite_path': '/home/teuthworker/src/github.com_Matan-B_ceph_729869e4a72e72e756b33d15b6f2fbf113fa1a2d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/Matan-B/ceph.git', 'suite_sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d', 'targets': {'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP65EXxXwkBSVWjXgoG0j74AGak5Q+rkRAN+nMX7H7VzMMSc+GAKyqBuvv3Qq1V0m6FjaKh8VWFprIHSO0lHglw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '729869e4a72e72e756b33d15b6f2fbf113fa1a2d', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-19_10:13:57', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078'} 2023-12-19T10:46:01.468 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-19T10:46:01.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2023-12-19T10:46:01.585 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:01.585 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2023-12-19T10:46:01.642 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-19T10:46:01.643 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-19 10:45:57.002528175 +0000 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-19 10:40:39.474357007 +0000 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-19 10:40:39.474357007 +0000 2023-12-19T10:46:01.698 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2023-12-19 10:40:39.474357007 +0000 2023-12-19T10:46:01.698 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-19T10:46:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-19T10:46:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-19T10:46:01.767 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000201959 s, 2.5 MB/s 2023-12-19T10:46:01.769 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-19T10:46:01.824 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-19 10:45:57.003528158 +0000 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-19 10:40:39.973348414 +0000 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-19 10:40:39.973348414 +0000 2023-12-19T10:46:01.880 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2023-12-19 10:40:39.973348414 +0000 2023-12-19T10:46:01.881 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-19T10:46:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-19T10:46:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-19T10:46:01.949 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000207907 s, 2.5 MB/s 2023-12-19T10:46:01.950 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-19T10:46:02.007 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2023-12-19T10:46:02.063 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-19 10:45:57.003528158 +0000 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-19 10:40:40.490339511 +0000 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-19 10:40:40.490339511 +0000 2023-12-19T10:46:02.154 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2023-12-19 10:40:40.490339511 +0000 2023-12-19T10:46:02.154 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-19T10:46:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-19T10:46:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-19T10:46:02.185 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000209923 s, 2.4 MB/s 2023-12-19T10:46:02.186 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-19T10:46:02.251 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2023-12-19T10:46:02.308 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-19T10:46:02.308 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-19T10:46:02.308 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-19T10:46:02.308 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-19T10:46:02.308 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2023-12-19T10:46:02.309 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2023-12-19 10:45:57.003528158 +0000 2023-12-19T10:46:02.309 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2023-12-19 10:40:40.984331004 +0000 2023-12-19T10:46:02.309 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2023-12-19 10:40:40.984331004 +0000 2023-12-19T10:46:02.309 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2023-12-19 10:40:40.984331004 +0000 2023-12-19T10:46:02.309 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-19T10:46:02.388 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2023-12-19T10:46:02.388 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2023-12-19T10:46:02.388 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000177935 s, 2.9 MB/s 2023-12-19T10:46:02.389 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-19T10:46:02.448 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-19T10:46:02.448 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-19T10:46:02.448 INFO:tasks.ceph:Generating config... 2023-12-19T10:46:02.449 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-19T10:46:02.449 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-19T10:46:02.449 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] err to stderr = True 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] log flush on exit = True 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] log to file = False 2023-12-19T10:46:02.450 INFO:tasks.ceph:[osd] log to stderr = True 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-19T10:46:02.451 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] auth client required = none 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] auth cluster required = none 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] auth service required = none 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] auth supported = none 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] debug asok = 0/0 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] debug auth = 0/0 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-12-19T10:46:02.451 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug client = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug context = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug crush = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug filer = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-12-19T10:46:02.452 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug journal = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-12-19T10:46:02.453 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug mon = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug monc = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug ms = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug osd = 0/0 2023-12-19T10:46:02.454 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rados = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug refs = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug striper = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug timer = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] debug tp = 0/0 2023-12-19T10:46:02.455 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms client mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[global] ms service mode = crc 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-12-19T10:46:02.456 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-12-19T10:46:02.456 INFO:tasks.ceph:Setting up mon.a... 2023-12-19T10:46:02.457 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-19T10:46:02.546 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.keyring 2023-12-19T10:46:02.550 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-19T10:46:02.605 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T10:46:02.636 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.33')] 2023-12-19T10:46:02.636 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.33', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-12-19T10:46:02.637 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:02.637 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T10:46:02.698 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.33 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:02.789 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:02.788+0000 7f862f85ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:02.789 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:02.788+0000 7f862f85ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:02.788+0000 7f862f85ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: generated fsid c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:setting min_mon_release = pacific 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:epoch 0 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:fsid c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:02.790 INFO:teuthology.orchestra.run.smithi033.stdout:last_changed 2023-12-19T10:46:02.790053+0000 2023-12-19T10:46:02.791 INFO:teuthology.orchestra.run.smithi033.stdout:created 2023-12-19T10:46:02.790053+0000 2023-12-19T10:46:02.791 INFO:teuthology.orchestra.run.smithi033.stdout:min_mon_release 16 (pacific) 2023-12-19T10:46:02.791 INFO:teuthology.orchestra.run.smithi033.stdout:election_strategy: 1 2023-12-19T10:46:02.791 INFO:teuthology.orchestra.run.smithi033.stdout:0: [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon.a 2023-12-19T10:46:02.791 INFO:teuthology.orchestra.run.smithi033.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-12-19T10:46:02.792 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-19T10:46:02.807 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c9991943-6900-4eb6-b39c-47f9b99419ec... 2023-12-19T10:46:02.809 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-19T10:46:02.913 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2023-12-19T10:46:02.913 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = none 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.914 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2023-12-19T10:46:02.915 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = 172.21.15.33 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: auth client required = none 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: auth cluster required = none 2023-12-19T10:46:02.916 INFO:teuthology.orchestra.run.smithi033.stdout: auth service required = none 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug asok = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug bdev = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug buffer = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug civetweb = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug client = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug compressor = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug context = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug crush = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug crypto = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug dpdk = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug eventtrace = 0/0 2023-12-19T10:46:02.917 INFO:teuthology.orchestra.run.smithi033.stdout: debug filer = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug filestore = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug finisher = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug fuse = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug heartbeatmap = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug javaclient = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug journal = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug journaler = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug kstore = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug leveldb = 0/0 2023-12-19T10:46:02.918 INFO:teuthology.orchestra.run.smithi033.stdout: debug lockdep = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds balancer = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds locker = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds log = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds log expire = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mds migrator = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug memdb = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgrc = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2023-12-19T10:46:02.919 INFO:teuthology.orchestra.run.smithi033.stdout: debug objclass = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug objectcacher = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug objecter = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug optracker = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug perfcounter = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug rados = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd mirror = 0/0 2023-12-19T10:46:02.920 INFO:teuthology.orchestra.run.smithi033.stdout: debug rbd replay = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug refs = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug rgw sync = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug striper = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug throttle = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug timer = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: debug tp = 0/0 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: ms client mode = crc 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: ms cluster mode = crc 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon client mode = crc 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon cluster mode = crc 2023-12-19T10:46:02.921 INFO:teuthology.orchestra.run.smithi033.stdout: ms mon service mode = crc 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: ms service mode = crc 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2023-12-19T10:46:02.922 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.923 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression algorithm = snappy 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore compression mode = aggressive 2023-12-19T10:46:02.924 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore rocksdb cf = False 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: crimson seastar smp = 3 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 20 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 20 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug filestore = 0/0 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug journal = 0/0 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 0/0 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: err to stderr = True 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: log flush on exit = True 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: log to file = False 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: log to stderr = True 2023-12-19T10:46:02.925 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2023-12-19T10:46:02.926 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 0/0 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 0/0 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 0/0 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2023-12-19T10:46:02.927 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2023-12-19T10:46:02.928 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2023-12-19T10:46:02.928 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-19T10:46:02.929 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default crimson = True 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-19T10:46:02.930 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2023-12-19T10:46:02.931 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-19T10:46:02.932 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-19T10:46:03.001 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-19T10:46:03.001 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:03.002 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-19T10:46:03.019 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:03.019 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-19T10:46:03.082 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:03.083 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:03.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-19T10:46:03.083 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-19T10:46:03.174 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:03.174 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:03.236 INFO:tasks.ceph:Setting up mon nodes... 2023-12-19T10:46:03.236 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-19T10:46:03.236 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T10:46:03.343 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-19T10:46:03.347 INFO:tasks.ceph:Setting up mds nodes... 2023-12-19T10:46:03.347 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-19T10:46:03.347 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-19T10:46:03.399 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-19T10:46:03.418 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-19T10:46:03.418 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-19T10:46:03.418 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-19T10:46:03.449 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-19T10:46:03.449 INFO:tasks.ceph:role: osd.0 2023-12-19T10:46:03.449 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:03.449 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T10:46:03.528 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-19T10:46:03.529 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T10:46:03.529 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T10:46:04.036 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-19T10:46:04.038 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-19T10:46:04.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-19T10:46:04.185 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-19T10:46:04.227 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-19T10:46:04.302 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-19T10:46:04.302 INFO:tasks.ceph:role: osd.1 2023-12-19T10:46:04.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:04.302 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-19T10:46:04.380 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T10:46:04.381 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T10:46:04.821 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-19T10:46:04.824 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-19T10:46:04.824 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-19T10:46:04.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-19T10:46:04.963 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-19T10:46:05.036 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-12-19T10:46:05.036 INFO:tasks.ceph:role: osd.2 2023-12-19T10:46:05.036 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:05.037 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2023-12-19T10:46:05.118 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-19T10:46:05.119 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-19T10:46:05.119 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-19T10:46:05.119 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-19T10:46:05.615 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2023-12-19T10:46:05.617 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2023-12-19T10:46:05.618 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-19T10:46:05.752 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-19T10:46:05.788 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:06.345 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:06.346 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,347 seastar - Reactor backend: linux-aio 2023-12-19T10:46:06.363 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,362 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:06.363 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,362 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:06.363 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,362 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:06.363 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,362 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:06.364 INFO:teuthology.orchestra.run.smithi033.stderr:==80626==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:06.371 INFO:teuthology.orchestra.run.smithi033.stderr:==80626==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe9d370000; bottom 0x7f674fee1000; size: 0x00974d48f000 (649836687360) 2023-12-19T10:46:06.371 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:06.371 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:06.927 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:06.980 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,980 seastar - Reactor backend: linux-aio 2023-12-19T10:46:06.998 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,998 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:06.998 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,998 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:06.998 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:06,998 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:07.005 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:07,006 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:07.008 INFO:teuthology.orchestra.run.smithi033.stderr:==80624==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:==80624==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe9d370000; bottom 0x7f674dcda000; size: 0x00974f696000 (649872367616) 2023-12-19T10:46:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:07.014 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:07.101 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbaab08 0x2cbabeb6 0x2cbad404 0x2cbad788 0x2cbad8d2 0x2cbadd28 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2670f1f8 0x26711841 0x26713652 0x26715eb7 0x265469df 0x1efab658 0x1ef1d065 0x1ef20ea6 0x2cf2f415 2023-12-19T10:46:07.101 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:11.070 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:11.110 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-19T10:46:11.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:11.701 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:11.701 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:11,699 seastar - Reactor backend: linux-aio 2023-12-19T10:46:11.718 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:11,718 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:11.719 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:11,718 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:11.719 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:11,718 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:11.719 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:11,719 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:11.721 INFO:teuthology.orchestra.run.smithi033.stderr:==80808==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:==80808==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff88753000; bottom 0x7f60bb245000; size: 0x009ecd50e000 (682049462272) 2023-12-19T10:46:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:11.727 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:11.753 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbaab08 0x2cbabeb6 0x2cbad404 0x2cbad788 0x2cbad8d2 0x2cbadd28 0x54daf 0x230c8f8b 0x23172983 0x231737b9 0x231748d6 0x23152e09 0x231534c0 0x2316365b 0x231657ea 0x231b64d4 0x23179e65 0x2317d48e 0x2319147f 0x231b1b31 0x231b23b9 0x2315f579 0x2316eb99 0x23170295 0x23170c6e 0x23171d59 0x232546b8 0x230b89b9 0x23182be4 0x2318462f 0x23185398 0x2314d204 0x2314c85c 0x22e590aa 0x24ea3960 0x24eab116 0x24eada64 0x24eaec3d 0x229f6257 0x22a46eb6 0x22a6a991 0x2cb4ce23 0x2cbeac30 0x2ce0f7a1 0x2ce115eb 0x2c865ce0 0x2c867be9 0x1f076374 0x1f07f111 0x1eeed53a 0x3feaf 0x3ff5f 0x1e996a44 2023-12-19T10:46:11.753 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2023-12-19T10:46:12.262 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:12.263 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:12.263 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:12.263 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:12.263 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:12.263 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:12.313 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:12.314 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:12,314 seastar - Reactor backend: linux-aio 2023-12-19T10:46:12.332 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:12,332 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:12.332 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:12,332 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:12.332 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:12,332 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:12.339 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:12,340 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:12.342 INFO:teuthology.orchestra.run.smithi033.stderr:==80806==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:12.348 INFO:teuthology.orchestra.run.smithi033.stderr:==80806==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff88753000; bottom 0x7f60b903e000; size: 0x009ecf715000 (682085142528) 2023-12-19T10:46:12.348 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:12.348 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:12.374 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbaab08 0x2cbabeb6 0x2cbad404 0x2cbad788 0x2cbad8d2 0x2cbadd28 0x54daf 0x22fad36c 0x22fb2671 0x22fb7ce2 0x24ea442b 0x24eab116 0x24eada64 0x24eaec3d 0x229f6257 0x22a46eb6 0x22a6a991 0x2cb4ce23 0x2cbeac30 0x2ce0f7a1 0x2ce115eb 0x2c865ce0 0x2c867be9 0x1eeeedf0 0x3feaf 0x3ff5f 0x1e996a44 2023-12-19T10:46:12.374 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2023-12-19T10:46:16.382 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:16.382 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:16.382 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:16.382 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:16.383 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:16.383 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:16.383 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:16.421 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-19T10:46:16.452 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:17.002 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:17.003 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,003 seastar - Reactor backend: linux-aio 2023-12-19T10:46:17.023 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,023 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:17.023 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,023 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:17.023 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,023 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:17.024 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,023 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:17.025 INFO:teuthology.orchestra.run.smithi033.stderr:==80990==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:17.032 INFO:teuthology.orchestra.run.smithi033.stderr:==80990==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff82605000; bottom 0x7f95cde99000; size: 0x0069b476c000 (453999247360) 2023-12-19T10:46:17.032 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:17.032 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:17.574 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:17.625 INFO:teuthology.orchestra.run.smithi033.stderr:WARNING: debug mode. Not for benchmarking or production 2023-12-19T10:46:17.626 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,626 seastar - Reactor backend: linux-aio 2023-12-19T10:46:17.654 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,653 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-12-19T10:46:17.654 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,653 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-12-19T10:46:17.654 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,654 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-12-19T10:46:17.661 INFO:teuthology.orchestra.run.smithi033.stderr:INFO 2023-12-19 10:46:17,661 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-12-19T10:46:17.663 INFO:teuthology.orchestra.run.smithi033.stderr:==80988==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-12-19T10:46:17.670 INFO:teuthology.orchestra.run.smithi033.stderr:==80988==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff82605000; bottom 0x7f95cbc92000; size: 0x0069b6973000 (454034927616) 2023-12-19T10:46:17.670 INFO:teuthology.orchestra.run.smithi033.stderr:False positive error reports may follow 2023-12-19T10:46:17.670 INFO:teuthology.orchestra.run.smithi033.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-12-19T10:46:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbaab08 0x2cbabeb6 0x2cbad404 0x2cbad788 0x2cbad8d2 0x2cbadd28 0x54daf 0x1ee280ab 0x22fa6c9b 0x22fad5db 0x22fb2671 0x22fb7ce2 0x24ea442b 0x24eab116 0x24eada64 0x24eaec3d 0x229f6257 0x22a46eb6 0x22a6a991 0x2cb4ce23 0x2cbeac30 0x2ce0f7a1 0x2ce115eb 0x2c865ce0 0x2c867be9 0x1eeeedf0 0x3feaf 0x3ff5f 0x1e996a44 2023-12-19T10:46:17.696 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2023-12-19T10:46:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2cbaab08 0x2cbabeb6 0x2cbad404 0x2cbad788 0x2cbad8d2 0x2cbadd28 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2670f1f8 0x26711841 0x26713652 0x26715eb7 0x265469df 0x1efab658 0x1ef1d065 0x1ef20ea6 0x2cf2f415 2023-12-19T10:46:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:kernel callstack: 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr:Suppressions used: 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr: count bytes template 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr: 1 8 MallocExtension::Register 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr:----------------------------------------------------- 2023-12-19T10:46:21.724 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:21.725 INFO:teuthology.orchestra.run.smithi033.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid c9991943-6900-4eb6-b39c-47f9b99419ec 2023-12-19T10:46:21.763 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-19T10:46:21.794 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-19T10:46:21.794 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:21.794 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-19T10:46:21.865 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:21.866 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-19T10:46:21.936 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:21.936 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-19T10:46:22.007 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:22.007 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-19T10:46:22.078 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:22.078 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-19T10:46:22.134 INFO:tasks.ceph:Adding keys to all mons... 2023-12-19T10:46:22.137 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBrdIFlM9uDFBAACcfDRXnqcyPo+65hvMwLfQ== 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBvdIFlIvvhAxAAYPyVJAb8usiHM7j+0tS0Cg== 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQB0dIFlCyO2FxAA5d9FHQ17m5LUJhyFnzs/zA== 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQB5dIFlxFPrKhAA+h5pFZKiVQeNZrB/Num2GQ== 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2023-12-19T10:46:22.203 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQBrdIFluwjZFxAAIFvDUAEKt+FRQEZZgbUM9g== 2023-12-19T10:46:22.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-19T10:46:22.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T10:46:22.370 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T10:46:22.432 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-19T10:46:22.494 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-19T10:46:22.559 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-19T10:46:22.559 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-19T10:46:22.589 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-19T10:46:22.697 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:22.696+0000 7f8d0623b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.697 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:22.696+0000 7f8d0623b980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.715 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-19T10:46:22.745 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-19T10:46:22.801 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-19T10:46:22.802 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-19T10:46:22.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-19T10:46:22.844 INFO:tasks.ceph.mon.a:Started 2023-12-19T10:46:22.844 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-19T10:46:22.844 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-19T10:46:22.845 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-19T10:46:22.846 INFO:tasks.ceph.mgr.x:Started 2023-12-19T10:46:22.847 INFO:tasks.ceph:Setting crush tunables to default 2023-12-19T10:46:22.847 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd crush tunables default 2023-12-19T10:46:22.931 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-12-19T10:46:22.930+0000 7f37bf60d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.931 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-12-19T10:46:22.930+0000 7f37bf60d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.931 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-12-19T10:46:22.930+0000 7f37bf60d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.931 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:22.931+0000 7fa3ce76af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.962 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:22.955+0000 7fc184d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:22.996 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:22.995+0000 7fc184d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:23.067 INFO:teuthology.orchestra.run.smithi033.stderr:adjusted tunables profile to default 2023-12-19T10:46:23.078 INFO:tasks.ceph:check_enable_crimson: True 2023-12-19T10:46:23.078 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-12-19T10:46:23.078 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-12-19T10:46:23.135 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:23.134+0000 7fa3ce76af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:23.135 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:23.134+0000 7fa3ce76af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:23.190 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:23.185+0000 7fea4ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:23.192 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:23.191+0000 7fea4ae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:23.272 INFO:tasks.ceph.mon.a.smithi033.stderr:2023-12-19T10:46:23.272+0000 7f37b75f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-12-19T10:46:23.273 INFO:tasks.ceph.mon.a.smithi033.stderr:Please be aware that this feature is experimental, untested, 2023-12-19T10:46:23.273 INFO:tasks.ceph.mon.a.smithi033.stderr:unsupported, and may result in data corruption, data loss, 2023-12-19T10:46:23.273 INFO:tasks.ceph.mon.a.smithi033.stderr:and/or irreparable damage to your cluster. Do not use 2023-12-19T10:46:23.273 INFO:tasks.ceph.mon.a.smithi033.stderr:feature with important data. 2023-12-19T10:46:23.273 INFO:tasks.ceph.mon.a.smithi033.stderr: 2023-12-19T10:46:23.614 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:23.613+0000 7fa3ce76af00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-19T10:46:23.668 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:23.667+0000 7fa3ce76af00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-19T10:46:24.080 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-19T10:46:24.080 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:24.080 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-19T10:46:24.111 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:24.111 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-19T10:46:24.182 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:24.182 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-19T10:46:24.223 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.222+0000 7fa3ce76af00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-19T10:46:24.253 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 1933045b-f7e5-42a7-b0e9-5606c9efdd9f 0 2023-12-19T10:46:24.278 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.277+0000 7fa3ce76af00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-19T10:46:24.326 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.325+0000 7fa3ce76af00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-19T10:46:24.384 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.383+0000 7fa3ce76af00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-19T10:46:24.388 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.383+0000 7fe09c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.393 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.390+0000 7fe09c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.430 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.429+0000 7fa3ce76af00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-19T10:46:24.468 INFO:teuthology.orchestra.run.smithi033.stdout:0 2023-12-19T10:46:24.480 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new da7d4c1d-ac13-4e29-9bc4-e3053836c709 1 2023-12-19T10:46:24.538 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.537+0000 7fa3ce76af00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-19T10:46:24.591 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.585+0000 7f2396d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.593 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.592+0000 7f2396d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.615 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.614+0000 7fa3ce76af00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-19T10:46:24.663 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.662+0000 7fa3ce76af00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-19T10:46:24.671 INFO:teuthology.orchestra.run.smithi033.stdout:1 2023-12-19T10:46:24.682 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new a78fc133-ec96-43a4-b79d-eb7807d0a609 2 2023-12-19T10:46:24.720 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.719+0000 7fa3ce76af00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-19T10:46:24.771 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:24.770+0000 7fa3ce76af00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-19T10:46:24.793 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.788+0000 7fa50af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.795 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:24.795+0000 7fa50af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:24.874 INFO:teuthology.orchestra.run.smithi033.stdout:2 2023-12-19T10:46:24.884 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-12-19T10:46:24.960 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-19T10:46:24.961 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-12-19T10:46:25.003 INFO:tasks.ceph.osd.0:Started 2023-12-19T10:46:25.003 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-12-19T10:46:25.038 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-19T10:46:25.038 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-12-19T10:46:25.081 INFO:tasks.ceph.osd.1:Started 2023-12-19T10:46:25.081 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-12-19T10:46:25.121 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-19T10:46:25.121 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-12-19T10:46:25.163 INFO:tasks.ceph.osd.2:Started 2023-12-19T10:46:25.164 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T10:46:25.199 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.198+0000 7fa3ce76af00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-19T10:46:25.301 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:25.296+0000 7f153f4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:25.304 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:25.303+0000 7f153f4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:25.377 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:25.377 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":6,"fsid":"c9991943-6900-4eb6-b39c-47f9b99419ec","created":"2023-12-19T10:46:22.988380+0000","modified":"2023-12-19T10:46:24.873465+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T10:46:24.873465+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1933045b-f7e5-42a7-b0e9-5606c9efdd9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da7d4c1d-ac13-4e29-9bc4-e3053836c709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a78fc133-ec96-43a4-b79d-eb7807d0a609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T10:46:25.390 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.389+0000 7fa3ce76af00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-19T10:46:25.394 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-19T10:46:25.394 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-19T10:46:25.527 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.526+0000 7fa3ce76af00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-19T10:46:25.585 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.584+0000 7fa3ce76af00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-19T10:46:25.695 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.694+0000 7fa3ce76af00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-19T10:46:25.696 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T10:46:25.797 INFO:teuthology.misc.health.smithi033.stderr:2023-12-19T10:46:25.793+0000 7fa849b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:25.799 INFO:teuthology.misc.health.smithi033.stderr:2023-12-19T10:46:25.798+0000 7fa849b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:25.805 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.804+0000 7fa3ce76af00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-19T10:46:25.866 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-19T10:46:25.867 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":6,"fsid":"c9991943-6900-4eb6-b39c-47f9b99419ec","created":"2023-12-19T10:46:22.988380+0000","modified":"2023-12-19T10:46:24.873465+0000","last_up_change":"0.000000","last_in_change":"2023-12-19T10:46:24.873465+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1933045b-f7e5-42a7-b0e9-5606c9efdd9f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"da7d4c1d-ac13-4e29-9bc4-e3053836c709","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a78fc133-ec96-43a4-b79d-eb7807d0a609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T10:46:25.883 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-19T10:46:25.966 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:25.965+0000 7fa3ce76af00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-19T10:46:26.014 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:26.013+0000 7fa3ce76af00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-19T10:46:26.170 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:26.170+0000 7fa3ce76af00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-19T10:46:27.037 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:27.037+0000 7fa3ce76af00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-19T10:46:27.325 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:27.325+0000 7fa3ce76af00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-19T10:46:27.380 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:27.380+0000 7fa3ce76af00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-19T10:46:27.488 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:27.488+0000 7fa3ce76af00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-19T10:46:28.509 INFO:tasks.ceph.mgr.x.smithi033.stderr:2023-12-19T10:46:28.508+0000 7fa36e4ab640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-19T10:46:32.185 DEBUG:teuthology.orchestra.run.smithi033:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-19T10:46:32.296 INFO:teuthology.misc.health.smithi033.stderr:2023-12-19T10:46:32.292+0000 7f9a3e79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:32.298 INFO:teuthology.misc.health.smithi033.stderr:2023-12-19T10:46:32.297+0000 7f9a3e79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:32.487 INFO:teuthology.misc.health.smithi033.stdout: 2023-12-19T10:46:32.487 INFO:teuthology.misc.health.smithi033.stdout:{"epoch":8,"fsid":"c9991943-6900-4eb6-b39c-47f9b99419ec","created":"2023-12-19T10:46:22.988380+0000","modified":"2023-12-19T10:46:30.509668+0000","last_up_change":"2023-12-19T10:46:30.509668+0000","last_in_change":"2023-12-19T10:46:24.873465+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"1933045b-f7e5-42a7-b0e9-5606c9efdd9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":4082017752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6803","nonce":4082017752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":4082017752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4082017752}]},"public_addr":"172.21.15.33:6802/4082017752","cluster_addr":"172.21.15.33:6803/4082017752","heartbeat_back_addr":"172.21.15.33:6809/4082017752","heartbeat_front_addr":"172.21.15.33:6808/4082017752","state":["exists","up"]},{"osd":1,"uuid":"da7d4c1d-ac13-4e29-9bc4-e3053836c709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2375922966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2375922966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2375922966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2375922966}]},"public_addr":"172.21.15.33:6804/2375922966","cluster_addr":"172.21.15.33:6805/2375922966","heartbeat_back_addr":"172.21.15.33:6811/2375922966","heartbeat_front_addr":"172.21.15.33:6810/2375922966","state":["exists","up"]},{"osd":2,"uuid":"a78fc133-ec96-43a4-b79d-eb7807d0a609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":419290455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":419290455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":419290455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":419290455}]},"public_addr":"172.21.15.33:6806/419290455","cluster_addr":"172.21.15.33:6807/419290455","heartbeat_back_addr":"172.21.15.33:6813/419290455","heartbeat_front_addr":"172.21.15.33:6812/419290455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T10:46:32.498 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-19T10:46:32.498 INFO:tasks.ceph:Creating RBD pool 2023-12-19T10:46:32.498 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-19T10:46:32.613 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:32.607+0000 7fbe3619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:32.615 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:32.614+0000 7fbe3619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:33.526 INFO:teuthology.orchestra.run.smithi033.stderr:pool 'rbd' created 2023-12-19T10:46:33.543 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-19T10:46:33.669 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:33.659+0000 7fb9e519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:33.671 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:33.670+0000 7fb9e519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:34.531 INFO:teuthology.orchestra.run.smithi033.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-19T10:46:34.544 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-19T10:46:34.544 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-19T10:46:34.544 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-19T10:46:34.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-19T10:46:34.545 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-19T10:46:34.678 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:34.674+0000 7f0c5a75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:34.679 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:34.679+0000 7f0c5a75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:34.920 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:34.931 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":1800873440},{"type":"v1","addr":"172.21.15.33:6801","nonce":1800873440}]},"active_addr":"172.21.15.33:6801/1800873440","active_change":"2023-12-19T10:46:27.501808+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1671396892}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":2995001789}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.33:0","nonce":1514376109}]}]} 2023-12-19T10:46:34.934 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-19T10:46:34.935 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-19T10:46:34.935 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T10:46:35.047 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.043+0000 7fc1c546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.049 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.048+0000 7fc1c546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.234 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:35.234 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":11,"fsid":"c9991943-6900-4eb6-b39c-47f9b99419ec","created":"2023-12-19T10:46:22.988380+0000","modified":"2023-12-19T10:46:34.526679+0000","last_up_change":"2023-12-19T10:46:30.509668+0000","last_in_change":"2023-12-19T10:46:24.873465+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T10:46:31.562028+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-19T10:46:32.826514+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1933045b-f7e5-42a7-b0e9-5606c9efdd9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":4082017752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6803","nonce":4082017752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":4082017752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4082017752}]},"public_addr":"172.21.15.33:6802/4082017752","cluster_addr":"172.21.15.33:6803/4082017752","heartbeat_back_addr":"172.21.15.33:6809/4082017752","heartbeat_front_addr":"172.21.15.33:6808/4082017752","state":["exists","up"]},{"osd":1,"uuid":"da7d4c1d-ac13-4e29-9bc4-e3053836c709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2375922966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2375922966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2375922966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2375922966}]},"public_addr":"172.21.15.33:6804/2375922966","cluster_addr":"172.21.15.33:6805/2375922966","heartbeat_back_addr":"172.21.15.33:6811/2375922966","heartbeat_front_addr":"172.21.15.33:6810/2375922966","state":["exists","up"]},{"osd":2,"uuid":"a78fc133-ec96-43a4-b79d-eb7807d0a609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":419290455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":419290455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":419290455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":419290455}]},"public_addr":"172.21.15.33:6806/419290455","cluster_addr":"172.21.15.33:6807/419290455","heartbeat_back_addr":"172.21.15.33:6813/419290455","heartbeat_front_addr":"172.21.15.33:6812/419290455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T10:46:35.248 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-19T10:46:35.248 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-19T10:46:35.360 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.356+0000 7f3817b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.362 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.361+0000 7f3817b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.547 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:35.547 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":12,"fsid":"c9991943-6900-4eb6-b39c-47f9b99419ec","created":"2023-12-19T10:46:22.988380+0000","modified":"2023-12-19T10:46:35.531193+0000","last_up_change":"2023-12-19T10:46:30.509668+0000","last_in_change":"2023-12-19T10:46:24.873465+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-19T10:46:31.562028+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-19T10:46:32.826514+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1933045b-f7e5-42a7-b0e9-5606c9efdd9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":4082017752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6803","nonce":4082017752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6809","nonce":4082017752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4082017752}]},"public_addr":"172.21.15.33:6802/4082017752","cluster_addr":"172.21.15.33:6803/4082017752","heartbeat_back_addr":"172.21.15.33:6809/4082017752","heartbeat_front_addr":"172.21.15.33:6808/4082017752","state":["exists","up"]},{"osd":1,"uuid":"da7d4c1d-ac13-4e29-9bc4-e3053836c709","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":2375922966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6805","nonce":2375922966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6811","nonce":2375922966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":2375922966}]},"public_addr":"172.21.15.33:6804/2375922966","cluster_addr":"172.21.15.33:6805/2375922966","heartbeat_back_addr":"172.21.15.33:6811/2375922966","heartbeat_front_addr":"172.21.15.33:6810/2375922966","state":["exists","up"]},{"osd":2,"uuid":"a78fc133-ec96-43a4-b79d-eb7807d0a609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":419290455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6807","nonce":419290455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6813","nonce":419290455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":419290455}]},"public_addr":"172.21.15.33:6806/419290455","cluster_addr":"172.21.15.33:6807/419290455","heartbeat_back_addr":"172.21.15.33:6813/419290455","heartbeat_front_addr":"172.21.15.33:6812/419290455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-19T10:46:35.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-19T10:46:35.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-19T10:46:35.561 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-19T10:46:35.684 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.679+0000 7f5e90cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.686 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.685+0000 7f5e90cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.689 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.684+0000 7fc179b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.690 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.684+0000 7fdb1059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.692 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.691+0000 7fc179b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.692 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.692+0000 7fdb1059e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.715 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:35.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T10:46:35.717 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:35.717 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-19T10:46:35.718 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:35.718 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-19T10:46:35.836 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.831+0000 7f42e222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.836 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.831+0000 7f05f3b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.836 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.831+0000 7f41e291e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.838 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.837+0000 7f05f3b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.839 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.837+0000 7f42e222d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:35.839 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:35.837+0000 7f41e291e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:36.026 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:36.032 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2023-12-19T10:46:36.032 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:36.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-12-19T10:46:36.040 DEBUG:teuthology.parallel:result is None 2023-12-19T10:46:36.046 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-19T10:46:36.047 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-12-19T10:46:36.047 DEBUG:teuthology.parallel:result is None 2023-12-19T10:46:37.047 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T10:46:37.175 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:37.170+0000 7f418f55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:37.177 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:37.177+0000 7f418f55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:37.363 INFO:teuthology.orchestra.run.smithi033.stdout:34359738372 2023-12-19T10:46:37.375 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-12-19T10:46:38.376 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-19T10:46:38.504 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:38.499+0000 7ff56ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:38.505 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:38.505+0000 7ff56ec2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:38.689 INFO:teuthology.orchestra.run.smithi033.stdout:34359738373 2023-12-19T10:46:38.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-12-19T10:46:38.701 DEBUG:teuthology.parallel:result is None 2023-12-19T10:46:38.701 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-19T10:46:38.702 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T10:46:38.814 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:38.810+0000 7f68f16ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:38.815 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:38.815+0000 7f68f16ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:38.998 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:38.998 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-19T10:46:39.010 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-19T10:46:37.506214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.804406"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.578549+0000","last_change":"2023-12-19T10:46:34.656971+0000","last_active":"2023-12-19T10:46:35.578549+0000","last_peered":"2023-12-19T10:46:35.578549+0000","last_clean":"2023-12-19T10:46:35.578549+0000","last_became_active":"2023-12-19T10:46:34.655160+0000","last_became_peered":"2023-12-19T10:46:34.655160+0000","last_unstale":"2023-12-19T10:46:35.578549+0000","last_undegraded":"2023-12-19T10:46:35.578549+0000","last_fullsized":"2023-12-19T10:46:35.578549+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.584869+0000","last_change":"2023-12-19T10:46:34.659561+0000","last_active":"2023-12-19T10:46:35.584869+0000","last_peered":"2023-12-19T10:46:35.584869+0000","last_clean":"2023-12-19T10:46:35.584869+0000","last_became_active":"2023-12-19T10:46:34.657792+0000","last_became_peered":"2023-12-19T10:46:34.657792+0000","last_unstale":"2023-12-19T10:46:35.584869+0000","last_undegraded":"2023-12-19T10:46:35.584869+0000","last_fullsized":"2023-12-19T10:46:35.584869+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.579389+0000","last_change":"2023-12-19T10:46:34.632958+0000","last_active":"2023-12-19T10:46:35.579389+0000","last_peered":"2023-12-19T10:46:35.579389+0000","last_clean":"2023-12-19T10:46:35.579389+0000","last_became_active":"2023-12-19T10:46:34.630708+0000","last_became_peered":"2023-12-19T10:46:34.630708+0000","last_unstale":"2023-12-19T10:46:35.579389+0000","last_undegraded":"2023-12-19T10:46:35.579389+0000","last_fullsized":"2023-12-19T10:46:35.579389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.583403+0000","last_change":"2023-12-19T10:46:34.631493+0000","last_active":"2023-12-19T10:46:35.583403+0000","last_peered":"2023-12-19T10:46:35.583403+0000","last_clean":"2023-12-19T10:46:35.583403+0000","last_became_active":"2023-12-19T10:46:34.624116+0000","last_became_peered":"2023-12-19T10:46:34.624116+0000","last_unstale":"2023-12-19T10:46:35.583403+0000","last_undegraded":"2023-12-19T10:46:35.583403+0000","last_fullsized":"2023-12-19T10:46:35.583403+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.578720+0000","last_change":"2023-12-19T10:46:34.620688+0000","last_active":"2023-12-19T10:46:35.578720+0000","last_peered":"2023-12-19T10:46:35.578720+0000","last_clean":"2023-12-19T10:46:35.578720+0000","last_became_active":"2023-12-19T10:46:34.618151+0000","last_became_peered":"2023-12-19T10:46:34.618151+0000","last_unstale":"2023-12-19T10:46:35.578720+0000","last_undegraded":"2023-12-19T10:46:35.578720+0000","last_fullsized":"2023-12-19T10:46:35.578720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.542049+0000","last_change":"2023-12-19T10:46:34.640097+0000","last_active":"2023-12-19T10:46:35.542049+0000","last_peered":"2023-12-19T10:46:35.542049+0000","last_clean":"2023-12-19T10:46:35.542049+0000","last_became_active":"2023-12-19T10:46:34.637320+0000","last_became_peered":"2023-12-19T10:46:34.637320+0000","last_unstale":"2023-12-19T10:46:35.542049+0000","last_undegraded":"2023-12-19T10:46:35.542049+0000","last_fullsized":"2023-12-19T10:46:35.542049+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.541352+0000","last_change":"2023-12-19T10:46:34.642453+0000","last_active":"2023-12-19T10:46:35.541352+0000","last_peered":"2023-12-19T10:46:35.541352+0000","last_clean":"2023-12-19T10:46:35.541352+0000","last_became_active":"2023-12-19T10:46:34.638461+0000","last_became_peered":"2023-12-19T10:46:34.638461+0000","last_unstale":"2023-12-19T10:46:35.541352+0000","last_undegraded":"2023-12-19T10:46:35.541352+0000","last_fullsized":"2023-12-19T10:46:35.541352+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.579885+0000","last_change":"2023-12-19T10:46:34.627097+0000","last_active":"2023-12-19T10:46:35.579885+0000","last_peered":"2023-12-19T10:46:35.579885+0000","last_clean":"2023-12-19T10:46:35.579885+0000","last_became_active":"2023-12-19T10:46:34.621756+0000","last_became_peered":"2023-12-19T10:46:34.621756+0000","last_unstale":"2023-12-19T10:46:35.579885+0000","last_undegraded":"2023-12-19T10:46:35.579885+0000","last_fullsized":"2023-12-19T10:46:35.579885+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.576435+0000","last_change":"2023-12-19T10:46:33.590380+0000","last_active":"2023-12-19T10:46:35.576435+0000","last_peered":"2023-12-19T10:46:35.576435+0000","last_clean":"2023-12-19T10:46:35.576435+0000","last_became_active":"2023-12-19T10:46:33.585381+0000","last_became_peered":"2023-12-19T10:46:33.585381+0000","last_unstale":"2023-12-19T10:46:35.576435+0000","last_undegraded":"2023-12-19T10:46:35.576435+0000","last_fullsized":"2023-12-19T10:46:35.576435+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:32.514775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:32.514775+0000","last_clean_scrub_stamp":"2023-12-19T10:46:32.514775+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-19T10:46:39.011 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-19T10:46:39.122 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:39.118+0000 7fca528ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:39.124 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:39.123+0000 7fca528ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:39.310 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:39.310 INFO:teuthology.orchestra.run.smithi033.stderr:dumped all 2023-12-19T10:46:39.321 INFO:teuthology.orchestra.run.smithi033.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-12-19T10:46:37.506214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59914,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.804406"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.578549+0000","last_change":"2023-12-19T10:46:34.656971+0000","last_active":"2023-12-19T10:46:35.578549+0000","last_peered":"2023-12-19T10:46:35.578549+0000","last_clean":"2023-12-19T10:46:35.578549+0000","last_became_active":"2023-12-19T10:46:34.655160+0000","last_became_peered":"2023-12-19T10:46:34.655160+0000","last_unstale":"2023-12-19T10:46:35.578549+0000","last_undegraded":"2023-12-19T10:46:35.578549+0000","last_fullsized":"2023-12-19T10:46:35.578549+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.584869+0000","last_change":"2023-12-19T10:46:34.659561+0000","last_active":"2023-12-19T10:46:35.584869+0000","last_peered":"2023-12-19T10:46:35.584869+0000","last_clean":"2023-12-19T10:46:35.584869+0000","last_became_active":"2023-12-19T10:46:34.657792+0000","last_became_peered":"2023-12-19T10:46:34.657792+0000","last_unstale":"2023-12-19T10:46:35.584869+0000","last_undegraded":"2023-12-19T10:46:35.584869+0000","last_fullsized":"2023-12-19T10:46:35.584869+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.579389+0000","last_change":"2023-12-19T10:46:34.632958+0000","last_active":"2023-12-19T10:46:35.579389+0000","last_peered":"2023-12-19T10:46:35.579389+0000","last_clean":"2023-12-19T10:46:35.579389+0000","last_became_active":"2023-12-19T10:46:34.630708+0000","last_became_peered":"2023-12-19T10:46:34.630708+0000","last_unstale":"2023-12-19T10:46:35.579389+0000","last_undegraded":"2023-12-19T10:46:35.579389+0000","last_fullsized":"2023-12-19T10:46:35.579389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.583403+0000","last_change":"2023-12-19T10:46:34.631493+0000","last_active":"2023-12-19T10:46:35.583403+0000","last_peered":"2023-12-19T10:46:35.583403+0000","last_clean":"2023-12-19T10:46:35.583403+0000","last_became_active":"2023-12-19T10:46:34.624116+0000","last_became_peered":"2023-12-19T10:46:34.624116+0000","last_unstale":"2023-12-19T10:46:35.583403+0000","last_undegraded":"2023-12-19T10:46:35.583403+0000","last_fullsized":"2023-12-19T10:46:35.583403+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.578720+0000","last_change":"2023-12-19T10:46:34.620688+0000","last_active":"2023-12-19T10:46:35.578720+0000","last_peered":"2023-12-19T10:46:35.578720+0000","last_clean":"2023-12-19T10:46:35.578720+0000","last_became_active":"2023-12-19T10:46:34.618151+0000","last_became_peered":"2023-12-19T10:46:34.618151+0000","last_unstale":"2023-12-19T10:46:35.578720+0000","last_undegraded":"2023-12-19T10:46:35.578720+0000","last_fullsized":"2023-12-19T10:46:35.578720+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.542049+0000","last_change":"2023-12-19T10:46:34.640097+0000","last_active":"2023-12-19T10:46:35.542049+0000","last_peered":"2023-12-19T10:46:35.542049+0000","last_clean":"2023-12-19T10:46:35.542049+0000","last_became_active":"2023-12-19T10:46:34.637320+0000","last_became_peered":"2023-12-19T10:46:34.637320+0000","last_unstale":"2023-12-19T10:46:35.542049+0000","last_undegraded":"2023-12-19T10:46:35.542049+0000","last_fullsized":"2023-12-19T10:46:35.542049+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.541352+0000","last_change":"2023-12-19T10:46:34.642453+0000","last_active":"2023-12-19T10:46:35.541352+0000","last_peered":"2023-12-19T10:46:35.541352+0000","last_clean":"2023-12-19T10:46:35.541352+0000","last_became_active":"2023-12-19T10:46:34.638461+0000","last_became_peered":"2023-12-19T10:46:34.638461+0000","last_unstale":"2023-12-19T10:46:35.541352+0000","last_undegraded":"2023-12-19T10:46:35.541352+0000","last_fullsized":"2023-12-19T10:46:35.541352+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.579885+0000","last_change":"2023-12-19T10:46:34.627097+0000","last_active":"2023-12-19T10:46:35.579885+0000","last_peered":"2023-12-19T10:46:35.579885+0000","last_clean":"2023-12-19T10:46:35.579885+0000","last_became_active":"2023-12-19T10:46:34.621756+0000","last_became_peered":"2023-12-19T10:46:34.621756+0000","last_unstale":"2023-12-19T10:46:35.579885+0000","last_undegraded":"2023-12-19T10:46:35.579885+0000","last_fullsized":"2023-12-19T10:46:35.579885+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:33.523119+0000","last_clean_scrub_stamp":"2023-12-19T10:46:33.523119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-19T10:46:35.576435+0000","last_change":"2023-12-19T10:46:33.590380+0000","last_active":"2023-12-19T10:46:35.576435+0000","last_peered":"2023-12-19T10:46:35.576435+0000","last_clean":"2023-12-19T10:46:35.576435+0000","last_became_active":"2023-12-19T10:46:33.585381+0000","last_became_peered":"2023-12-19T10:46:33.585381+0000","last_unstale":"2023-12-19T10:46:35.576435+0000","last_undegraded":"2023-12-19T10:46:35.576435+0000","last_fullsized":"2023-12-19T10:46:35.576435+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-19T10:46:32.514775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-19T10:46:32.514775+0000","last_clean_scrub_stamp":"2023-12-19T10:46:32.514775+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29957,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-12-19T10:46:39.322 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-19T10:46:39.322 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-19T10:46:39.322 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-19T10:46:39.323 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-19T10:46:39.433 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:39.430+0000 7f98f0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:39.435 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:46:39.434+0000 7f98f0a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:46:39.653 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:39.653 INFO:teuthology.orchestra.run.smithi033.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-19T10:46:39.665 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-19T10:46:39.665 INFO:teuthology.run_tasks:Running task cbt... 2023-12-19T10:46:39.679 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi033.front.sepia.ceph.com', 'osds': ['ubuntu@smithi033.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi033.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi033.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi033.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-12-19T10:46:39.679 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-12-19T10:46:39.697 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:46:39.697 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-19T10:46:39.749 DEBUG:teuthology.orchestra.run.smithi033:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-12-19T10:46:39.804 INFO:teuthology.orchestra.run.smithi033.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-12-19T10:46:40.192 INFO:tasks.cbt:Installing collectl 2023-12-19T10:46:40.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-12-19T10:46:40.297 INFO:teuthology.orchestra.run.smithi033.stderr:--2023-12-19 10:46:40-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-12-19T10:46:40.441 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2023-12-19T10:46:40.451 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-12-19T10:46:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-19T10:46:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlgXSQg61XSdF3-eGP5rU-ZmoSkcyrbPd5rD_jb5QOqh-P2PZw9_IDS1eJ8i6LT1bLDVj8h336TgzGYaq2FG-sLizCVg%3D%3D&use_mirror=versaweb&r= [following] 2023-12-19T10:46:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:--2023-12-19 10:46:40-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlgXSQg61XSdF3-eGP5rU-ZmoSkcyrbPd5rD_jb5QOqh-P2PZw9_IDS1eJ8i6LT1bLDVj8h336TgzGYaq2FG-sLizCVg%3D%3D&use_mirror=versaweb&r= 2023-12-19T10:46:40.678 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-12-19T10:46:40.749 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-12-19T10:46:40.990 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-19T10:46:40.990 INFO:teuthology.orchestra.run.smithi033.stderr:Location: https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-12-19T10:46:40.990 INFO:teuthology.orchestra.run.smithi033.stderr:--2023-12-19 10:46:40-- https://versaweb.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-12-19T10:46:41.002 INFO:teuthology.orchestra.run.smithi033.stderr:Resolving versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)... 162.251.232.173 2023-12-19T10:46:41.073 INFO:teuthology.orchestra.run.smithi033.stderr:Connecting to versaweb.dl.sourceforge.net (versaweb.dl.sourceforge.net)|162.251.232.173|:443... connected. 2023-12-19T10:46:41.369 INFO:teuthology.orchestra.run.smithi033.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-19T10:46:41.369 INFO:teuthology.orchestra.run.smithi033.stderr:Length: 620693 (606K) [application/x-gzip] 2023-12-19T10:46:41.369 INFO:teuthology.orchestra.run.smithi033.stderr:Saving to: ‘collectl.tar.gz’ 2023-12-19T10:46:41.369 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:41.443 INFO:teuthology.orchestra.run.smithi033.stderr: 0K .......... .......... .......... .......... .......... 8% 671K 1s 2023-12-19T10:46:41.511 INFO:teuthology.orchestra.run.smithi033.stderr: 50K .......... .......... .......... .......... .......... 16% 736K 1s 2023-12-19T10:46:41.514 INFO:teuthology.orchestra.run.smithi033.stderr: 100K .......... .......... .......... .......... .......... 24% 13.9M 0s 2023-12-19T10:46:41.581 INFO:teuthology.orchestra.run.smithi033.stderr: 150K .......... .......... .......... .......... .......... 32% 748K 0s 2023-12-19T10:46:41.586 INFO:teuthology.orchestra.run.smithi033.stderr: 200K .......... .......... .......... .......... .......... 41% 9.54M 0s 2023-12-19T10:46:41.588 INFO:teuthology.orchestra.run.smithi033.stderr: 250K .......... .......... .......... .......... .......... 49% 41.6M 0s 2023-12-19T10:46:41.589 INFO:teuthology.orchestra.run.smithi033.stderr: 300K .......... .......... .......... .......... .......... 57% 41.7M 0s 2023-12-19T10:46:41.650 INFO:teuthology.orchestra.run.smithi033.stderr: 350K .......... .......... .......... .......... .......... 65% 821K 0s 2023-12-19T10:46:41.655 INFO:teuthology.orchestra.run.smithi033.stderr: 400K .......... .......... .......... .......... .......... 74% 9.01M 0s 2023-12-19T10:46:41.662 INFO:teuthology.orchestra.run.smithi033.stderr: 450K .......... .......... .......... .......... .......... 82% 7.49M 0s 2023-12-19T10:46:41.666 INFO:teuthology.orchestra.run.smithi033.stderr: 500K .......... .......... .......... .......... .......... 90% 12.4M 0s 2023-12-19T10:46:41.675 INFO:teuthology.orchestra.run.smithi033.stderr: 550K .......... .......... .......... .......... .......... 98% 5.34M 0s 2023-12-19T10:46:41.675 INFO:teuthology.orchestra.run.smithi033.stderr: 600K ...... 100% 11.4T=0.3s 2023-12-19T10:46:41.675 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:41.675 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19 10:46:41 (1.93 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-12-19T10:46:41.675 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/ 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/collectl.conf 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/SlowProc.html 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/WhySummary.html 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Socket.html 2023-12-19T10:46:41.720 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/colmux.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Tutorial.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Gexpr.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Import.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/style.css 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/FileNaming.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/TheMath.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Environmental.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Process.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/HiResTime.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-12-19T10:46:41.721 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/slub.jpg 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Lustre.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Documentation.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Inodes.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Graphite.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/InputFiles.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Export.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Interrupts.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Startup.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Performance.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Messages.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Logging.html 2023-12-19T10:46:41.722 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-detail.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/CPUs.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Matrix.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-brief.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Misc.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Examples.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OpenStack.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Data.html 2023-12-19T10:46:41.723 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Exceptions.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Playback.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Hello.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Sockets.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Network.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Infiniband.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Memory.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/Disks.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-12-19T10:46:41.724 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/service/ 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/service/collectl.service 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/ 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-suse 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-debian 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl-generic 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/initd/collectl 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/misc.ph 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/gexpr.ph 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/statsd.ph 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/envrules.std 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/hello.ph 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vmsum.ph 2023-12-19T10:46:41.725 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/proctree.ph 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/README-WINDOWS 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/formatit.ph 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/colmux 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/INSTALL 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/graphite.ph 2023-12-19T10:46:41.726 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/RELEASE-collectl 2023-12-19T10:46:41.727 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vmstat.ph 2023-12-19T10:46:41.727 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/README 2023-12-19T10:46:41.727 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/ 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/collectl.1 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/man1/colmux.1 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/lexpr.ph 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/client.pl 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/collectl 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/COPYING 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/vnet.ph 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/GPL 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/UNINSTALL 2023-12-19T10:46:41.728 INFO:teuthology.orchestra.run.smithi033.stdout:collectl-4.3.1/ARTISTIC 2023-12-19T10:46:41.785 INFO:teuthology.orchestra.run.smithi033.stderr:./INSTALL: line 90: chkconfig: command not found 2023-12-19T10:46:41.787 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-12-19T10:46:42.371 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:03 ago on Tue 19 Dec 2023 10:41:39 AM UTC. 2023-12-19T10:46:42.470 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-12-19T10:46:42.473 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-12-19T10:46:42.531 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout: librbd-devel x86_64 2:19.0.0-231.g729869e4.el9 ceph 23 k 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-12-19T10:46:42.532 INFO:teuthology.orchestra.run.smithi033.stdout: perf x86_64 5.14.0-391.el9 appstream 7.6 M 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout: libradospp-devel x86_64 2:19.0.0-231.g729869e4.el9 ceph 30 k 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout:Install 5 Packages 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:42.533 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 7.8 M 2023-12-19T10:46:42.534 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 11 M 2023-12-19T10:46:42.534 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2023-12-19T10:46:42.829 INFO:teuthology.orchestra.run.smithi033.stdout:(1/5): librbd-devel-19.0.0-231.g729869e4.el9.x8 139 kB/s | 23 kB 00:00 2023-12-19T10:46:42.845 INFO:teuthology.orchestra.run.smithi033.stdout:(2/5): libradospp-devel-19.0.0-231.g729869e4.el 165 kB/s | 30 kB 00:00 2023-12-19T10:46:43.004 INFO:teuthology.orchestra.run.smithi033.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 83 kB/s | 13 kB 00:00 2023-12-19T10:46:43.029 INFO:teuthology.orchestra.run.smithi033.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 591 kB/s | 118 kB 00:00 2023-12-19T10:46:44.680 INFO:teuthology.orchestra.run.smithi033.stdout:(5/5): perf-5.14.0-391.el9.x86_64.rpm 3.8 MB/s | 7.6 MB 00:02 2023-12-19T10:46:44.683 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2023-12-19T10:46:44.683 INFO:teuthology.orchestra.run.smithi033.stdout:Total 3.6 MB/s | 7.8 MB 00:02 2023-12-19T10:46:44.768 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2023-12-19T10:46:44.792 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2023-12-19T10:46:44.792 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2023-12-19T10:46:45.004 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2023-12-19T10:46:45.005 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2023-12-19T10:46:45.410 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2023-12-19T10:46:45.493 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-12-19T10:46:45.529 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-12-19T10:46:45.560 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradospp-devel-2:19.0.0-231.g729869e4.el9.x86_64 3/5 2023-12-19T10:46:45.914 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librbd-devel-2:19.0.0-231.g729869e4.el9.x86_64 4/5 2023-12-19T10:46:46.043 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : perf-5.14.0-391.el9.x86_64 5/5 2023-12-19T10:46:47.012 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: perf-5.14.0-391.el9.x86_64 5/5 2023-12-19T10:46:47.012 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradospp-devel-2:19.0.0-231.g729869e4.el9.x86_64 1/5 2023-12-19T10:46:47.012 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd-devel-2:19.0.0-231.g729869e4.el9.x86_64 2/5 2023-12-19T10:46:47.013 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : perf-5.14.0-391.el9.x86_64 3/5 2023-12-19T10:46:47.013 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-12-19T10:46:47.362 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: libradospp-devel-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: librbd-devel-2:19.0.0-231.g729869e4.el9.x86_64 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-2.34-7.el9.x86_64 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: perf-5.14.0-391.el9.x86_64 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout: 2023-12-19T10:46:47.363 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2023-12-19T10:46:47.755 INFO:tasks.cbt:benchmark: radosbench 2023-12-19T10:46:47.755 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-12-19T10:46:47.788 DEBUG:teuthology.orchestra.run.smithi033:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-12-19T10:46:48.166 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - Settings.cluster: 2023-12-19T10:46:48.166 INFO:teuthology.orchestra.run.smithi033.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-12-19T10:46:48.166 INFO:teuthology.orchestra.run.smithi033.stderr: 'clients': ['ubuntu@smithi033.front.sepia.ceph.com'], 2023-12-19T10:46:48.166 INFO:teuthology.orchestra.run.smithi033.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'head': 'ubuntu@smithi033.front.sepia.ceph.com', 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'is_teuthology': True, 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'iterations': 1, 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'mons': ['ubuntu@smithi033.front.sepia.ceph.com'], 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'osds': ['ubuntu@smithi033.front.sepia.ceph.com'], 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'osds_per_node': 3, 2023-12-19T10:46:48.167 INFO:teuthology.orchestra.run.smithi033.stderr: 'pid_dir': '/var/run/ceph', 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'pgp_size': 256, 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'replication': 'replicated'}}, 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'rebuild_every_test': False, 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'rgws': {'ubuntu@smithi033.front.sepia.ceph.com': None}, 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'tmp_dir': '/tmp/cbt', 2023-12-19T10:46:48.168 INFO:teuthology.orchestra.run.smithi033.stderr: 'use_existing': True, 2023-12-19T10:46:48.169 INFO:teuthology.orchestra.run.smithi033.stderr: 'user': 'ubuntu'} 2023-12-19T10:46:48.187 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-12-19T10:46:48.188 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.189 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.189 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2023-12-19T10:46:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.192 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-19T10:46:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.194 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2023-12-19T10:46:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:46:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-19T10:46:48.197 INFO:teuthology.orchestra.run.smithi033.stderr:10:46:48 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-19T10:47:48.225 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.225 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-19T10:47:48.257 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.257 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-19T10:47:48.296 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.296 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-12-19T10:47:48.298 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.298 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-12-19T10:47:48.299 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.300 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.300 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2023-12-19T10:47:48.322 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.322 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi033.front.sepia.ceph.com ; done 2023-12-19T10:47:48.326 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.326 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.326 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-12-19T10:47:48.350 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - WARNING - cbt - sudo killall -9 rados 2023-12-19T10:47:48.350 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - WARNING - cbt - error 1 seen, continuing anyway... 2023-12-19T10:47:48.369 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2023-12-19T10:47:48.370 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-12-19T10:47:48.370 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.370 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.370 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-12-19T10:47:48.389 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.389 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.390 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2023-12-19T10:47:48.392 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.392 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-19T10:47:48.394 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.394 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.394 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2023-12-19T10:47:48.397 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.397 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-19T10:47:48.397 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-19T10:47:48.397 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.397 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.398 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-12-19T10:47:48.700 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:48.701 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-12-19T10:47:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:49 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:49 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:49.039 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-12-19T10:47:50.044 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:50 - INFO - cbt - Checking Health after pool creation. 2023-12-19T10:47:55.050 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:55 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:47:55.050 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:55 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:47:55.050 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-19T10:47:55.393 INFO:teuthology.orchestra.run.smithi033.stderr:10:47:55 - INFO - cbt - Final Pool Health Check. 2023-12-19T10:48:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.399 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.400 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-12-19T10:48:00.717 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.717 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-19T10:48:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.734 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-19T10:48:00.766 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.766 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-12-19T10:48:00.768 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.768 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-12-19T10:48:00.769 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.769 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:00.770 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-12-19T10:48:01.060 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:01.060 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-12-19T10:48:01.654 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:01.654 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:01.654 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-12-19T10:48:01.967 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:01.968 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:01.968 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2023-12-19T10:48:02.025 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.025 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.026 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2023-12-19T10:48:02.797 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.797 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.797 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2023-12-19T10:48:02.799 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.799 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-12-19T10:48:02.800 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.800 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.800 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2023-12-19T10:48:02.803 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.803 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - PerfMonitoring: in local_node 2023-12-19T10:48:02.804 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-12-19T10:48:02.804 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - INFO - cbt - Running radosbench write test. 2023-12-19T10:48:02.804 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.804 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.804 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2023-12-19T10:48:02.805 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.805 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:48:02.805 INFO:teuthology.orchestra.run.smithi033.stderr:10:48:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2023-12-19T10:49:03.921 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.922 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-12-19T10:49:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.946 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-12-19T10:49:03.979 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.980 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2023-12-19T10:49:03.981 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.981 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2023-12-19T10:49:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:03.983 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:03 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2023-12-19T10:49:04.292 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.304 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.304 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2023-12-19T10:49:04.324 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - get_fqdn_local()=smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.324 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi033.front.sepia.ceph.com ; done 2023-12-19T10:49:04.355 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - INFO - cbt - Convert results to json format. 2023-12-19T10:49:04.356 INFO:teuthology.orchestra.run.smithi033.stderr:10:49:04 - DEBUG - cbt - Nodes : ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.377 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-12-19T10:49:04.386 DEBUG:teuthology.orchestra.run.smithi033:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-12-19T10:49:04.571 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-12-19T10:49:04.571 INFO:tasks.cbt_performance:reading cbt preferences 2023-12-19T10:49:04.571 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-12-19T10:49:04.619 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.620 INFO:teuthology.orchestra.run.smithi033.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi033.front.sepia.ceph.com 2023-12-19T10:49:04.621 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:49:04.621 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi033.front.sepia.ceph.com of=/dev/stdout 2023-12-19T10:49:04.637 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-19T10:49:04.637 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-12-19T10:49:04.692 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-19T10:49:04.692 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-19T10:49:04.693 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2023-12-19T10:49:04.693 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi033.front.sepia.ceph.com of=/dev/stdout 2023-12-19T10:49:04.748 INFO:tasks.cbt_performance:reading total cpu cycles 2023-12-19T10:49:04.748 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-12-19T10:49:04.803 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-12-19T10:49:04.803 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-12-19T10:49:04.812 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-19T10:49:04.819 INFO:tasks.cbt_performance:Data inserted successfully. 2023-12-19T10:49:04.819 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-19T10:49:04.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-19T10:49:04.955 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:49:04.947+0000 7fc85242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:49:04.956 INFO:teuthology.orchestra.run.smithi033.stderr:2023-12-19T10:49:04.955+0000 7fc85242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-12-19T10:49:05.161 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-19T10:49:05.161 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-19T10:49:05.162 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-19T10:49:05.162 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T10:49:05.255 INFO:tasks.ceph.osd.0:Stopped 2023-12-19T10:49:05.255 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-19T10:49:05.255 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T10:49:08.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-19T10:49:11.356 INFO:tasks.ceph.osd.1:Stopped 2023-12-19T10:49:11.357 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-19T10:49:11.357 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T10:49:14.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-19T10:49:14.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-19T10:49:17.459 INFO:tasks.ceph.osd.2:Stopped 2023-12-19T10:49:17.459 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-19T10:49:17.459 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-19T10:49:17.460 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T10:49:17.491 INFO:tasks.ceph.mgr.x:Stopped 2023-12-19T10:49:17.491 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-19T10:49:17.491 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-19T10:49:17.491 INFO:teuthology.orchestra.run:waiting for 300 2023-12-19T10:49:17.548 INFO:tasks.ceph.mon.a:Stopped 2023-12-19T10:49:17.548 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-19T10:49:17.548 DEBUG:teuthology.orchestra.run.smithi033:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-12-19T10:49:17.650 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:17.650 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-19T10:49:18.043 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:18.043 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-19T10:49:18.139 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:18.140 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-19T10:49:18.205 INFO:tasks.ceph:Archiving mon data... 2023-12-19T10:49:18.206 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733/data/mon.a.tgz 2023-12-19T10:49:18.206 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2023-12-19T10:49:18.229 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.0CqlmOHWsj 2023-12-19T10:49:18.229 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0CqlmOHWsj 2023-12-19T10:49:18.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.0CqlmOHWsj 2023-12-19T10:49:18.467 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.0CqlmOHWsj is 178KB 2023-12-19T10:49:18.519 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.0CqlmOHWsj 2023-12-19T10:49:18.534 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-19T10:49:18.534 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-19T10:49:18.668 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-19T10:49:18.668 INFO:tasks.ceph:Archiving crash dumps... 2023-12-19T10:49:18.670 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733/remote/smithi033/crash 2023-12-19T10:49:18.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-19T10:49:18.738 INFO:tasks.ceph:Compressing logs... 2023-12-19T10:49:18.738 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-19T10:49:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-19T10:49:18.823 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82834.log 2023-12-19T10:49:18.824 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83797.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82384.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83936.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82963.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83849.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82921.log 2023-12-19T10:49:18.825 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82543.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82707.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80140.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84173.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83937.log 2023-12-19T10:49:18.826 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81975.log 2023-12-19T10:49:18.833 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81599.log 2023-12-19T10:49:18.833 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82709.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82835.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.83797.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82834.log: /var/log/ceph/ceph-client.admin.82707.log: /var/log/ceph/ceph.tmp-client.admin.80140.log: /var/log/ceph/ceph-client.admin.81975.log: /var/log/ceph/ceph-client.admin.81599.log: /var/log/ceph/ceph-client.admin.82384.log: /var/log/ceph/ceph-client.admin.83936.log: /var/log/ceph/ceph.audit.log: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83005.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.84173.log: /var/log/ceph/ceph-client.admin.83937.log: /var/log/ceph/ceph-client.admin.83849.log: /var/log/ceph/ceph-client.admin.82835.log: /var/log/ceph/ceph-client.admin.82709.log: 49.6% 65.6% -- replaced with /var/log/ceph/ceph-client.admin.83937.log.gz 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82921.log: /var/log/ceph/ceph-client.admin.82963.log: /var/log/ceph/ceph-client.admin.82543.log: -- replaced with /var/log/ceph/ceph-client.admin.84173.log.gz 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.1.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.83936.log.gz 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81947.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.83005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82585.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.81947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82708.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82440.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83089.log 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81599.log.gz 2023-12-19T10:49:18.834 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81511.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.83089.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82482.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.81511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81709.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82627.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip/var/log/ceph/ceph-client.admin.81709.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81789.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83826.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.81789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83047.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.83826.log: gzip/var/log/ceph/ceph-client.admin.83047.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82834.log.gz 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.x.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82835.log.gz 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.0.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82921.log.gz 2023-12-19T10:49:18.835 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82384.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83872.log 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82833.log 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 84.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81511.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.83872.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82440.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83735.log 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82482.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81749.log 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82543.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.82833.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82585.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.83735.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82627.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82708.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.82709.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.81749.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82707.log.gz 2023-12-19T10:49:18.836 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82833.log.gz 2023-12-19T10:49:18.837 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82963.log.gz 2023-12-19T10:49:18.838 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83005.log.gz 2023-12-19T10:49:18.838 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83047.log.gz 2023-12-19T10:49:18.838 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83089.log.gz 2023-12-19T10:49:18.838 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81975.log.gz 2023-12-19T10:49:18.841 INFO:teuthology.orchestra.run.smithi033.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80140.log.gz 2023-12-19T10:49:18.844 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81947.log.gz 2023-12-19T10:49:18.848 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83797.log.gz 2023-12-19T10:49:18.848 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83826.log.gz 2023-12-19T10:49:18.848 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83849.log.gz 2023-12-19T10:49:18.848 INFO:teuthology.orchestra.run.smithi033.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83872.log.gz 2023-12-19T10:49:18.869 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81709.log.gz 2023-12-19T10:49:18.870 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81749.log.gz 2023-12-19T10:49:18.870 INFO:teuthology.orchestra.run.smithi033.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81789.log.gz 2023-12-19T10:49:18.873 INFO:teuthology.orchestra.run.smithi033.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83735.log.gz 2023-12-19T10:49:18.948 INFO:teuthology.orchestra.run.smithi033.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-19T10:49:19.150 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-19T10:49:20.027 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-19T10:49:20.150 INFO:teuthology.orchestra.run.smithi033.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-19T10:49:20.177 INFO:teuthology.orchestra.run.smithi033.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-19T10:49:20.179 INFO:teuthology.orchestra.run.smithi033.stderr: 2023-12-19T10:49:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m1.386s 2023-12-19T10:49:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m2.782s 2023-12-19T10:49:20.179 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.144s 2023-12-19T10:49:20.179 INFO:tasks.ceph:Archiving logs... 2023-12-19T10:49:20.179 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733/remote/smithi033/log 2023-12-19T10:49:20.180 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-19T10:49:20.731 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-19T10:49:20.740 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-19T10:49:20.741 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-19T10:49:20.800 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-19T10:49:20.809 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-19T10:49:20.809 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-19T10:49:20.829 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2023-12-19T10:49:20.888 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-19T10:49:20.888 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2023-12-19T10:49:20.888 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +1066us[+1066us] +/- 74ms 2023-12-19T10:49:20.889 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv02.front.sepia.ceph.com 3 6 377 31 -782us[ -774us] +/- 54ms 2023-12-19T10:49:20.889 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +76us[ +76us] +/- 77ms 2023-12-19T10:49:20.889 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-19T10:49:20.894 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-19T10:49:20.903 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-19T10:49:20.904 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-19T10:49:20.951 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-19T10:49:21.133 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-19T10:49:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T10:49:21.169 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-19T10:49:21.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-19T10:49:21.212 INFO:teuthology.task.internal:Duration was 833.783315 seconds 2023-12-19T10:49:21.213 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-19T10:49:21.292 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-19T10:49:21.292 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-19T10:49:21.443 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-19T10:49:21.736 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-19T10:49:21.736 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2023-12-19T10:49:21.736 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-19T10:49:21.758 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-19T10:49:21.758 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-19T10:49:21.832 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-19T10:49:21.832 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-19T10:49:21.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-19T10:49:21.994 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-19T10:49:21.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-19T10:49:22.030 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-19T10:49:22.040 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-19T10:49:22.116 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2023-12-19T10:49:22.145 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-19T10:49:22.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-19T10:49:22.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-19T10:49:22.170 INFO:teuthology.task.internal:Transferring archived files... 2023-12-19T10:49:22.170 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-12-19_10:13:57-crimson-rados-wip-matanb-crimson-testing-misc-distro-crimson-smithi/7496733/remote/smithi033 2023-12-19T10:49:22.170 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-19T10:49:22.267 INFO:teuthology.task.internal:Removing archive directory... 2023-12-19T10:49:22.268 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-19T10:49:22.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-19T10:49:22.326 INFO:teuthology.task.internal:Not uploading archives. 2023-12-19T10:49:22.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-19T10:49:22.366 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-19T10:49:22.367 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-19T10:49:22.382 INFO:teuthology.orchestra.run.smithi033.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 19 10:49 /home/ubuntu/cephtest 2023-12-19T10:49:22.383 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-19T10:49:22.412 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-19T10:49:22.486 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-testing-misc&var-job_id_selected=7496733 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} duration: 833.783314704895 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-12-19T10:49:22.486 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-19T10:49:22.560 INFO:teuthology.run:pass